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Measure  Like  a  Fighter  Pilot 

When  the  late  Col.  John  Boyd  developed  the  Observe,  Orient, 
Decide,  and  Act  Loop  as  a  strategy  for  air  combat  and  warfare, 
he  probably  never  guessed  it  would  benefit  the  measurement  and 
analysis  process. 
by  Joe  H.  Eindly 


Applying  Functional  TSP  to  a  Maintenance  Project 

This  article  dispels  the  myth  that  the  Personal  Software  Process 
and  Team  Software  Process  only  work  for  new  code  development 
by  showing  how  one  organization  successfully  adapted  to  plan 
and  manage  a  maintenance  project. 
by  Ellen  George  and  Dr.  Steve  Janis^ewski 
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outstanding  performance. 
by  David  P.  Quinn 
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From  the  Publisher 

Integrated  Teams  and  Sound 
Processes  Bring  Success 

This  month,  CROSSTALK  is  pleased  to  publish  articles  from  the  five  winning  pro¬ 
grams  of  the  Top  5  Department  of  Defense  Program  Awards  2004  contest.  As  one 
of  the  judges  on  this  year’s  selection  committee,  I  was  delighted  to  see  that  many  of  the 
nominees  were  fine  examples  of  projects 
that  are  succeeding  at  developing,  sustaining, 
or  integrating  software  into  government  sys¬ 
tems  today.  Too  often  we  hear  of  projects 
that  are  failing.  It’s  refreshing  to  see  projects  that  are 
triumphant  given  continual  advances  in  software  tech¬ 
nology  along  with  the  challenging  goals  of  producing 
quality  software  faster,  better,  and  cheaper. 

A  common  theme  among  the  Top  5  winners  is 
having  an  integrated  team  that  continually  involves  its 
stakeholders.  Success  can  be  realized  when  customers, 
acquirers,  developers,  testers,  configuration  managers, 
end  users,  etc.  aU  work  together  throughout  the  sys¬ 
tem  life  cycle,  especially  during  requirements  analysis. 

Following  sound  processes  is  another  key  factor  to 
this  year’s  Top  5  winners’  success.  Whether  it  means 
employing  a  spiral  development  approach,  or  agile 
programming  practices,  or  Capability  Maturity  Model 
Integration  practices,  a  common  thread  is  to  imple¬ 
ment  and  follow  well-defined  processes  to  ensure 
quality  and  customer  satisfaction.  Congratulations  to 
aU  those  involved  in  these  programs. 

Continuing  on  this  theme  of  developing  software 
with  proven  practices,  a  caution  to  involve  management 
in  process  implementation  is  discussed  in  The  Myth  of 
the  Best  Practices  Silver  Bullet  by  Michael  W  Evans, 

Corinne  Segura,  and  Frank  Doherty.  These  authors  dis¬ 
cuss  why  it  is  critical  that  management  select  the  orga¬ 
nization’s  best  practices  and  understand  the  costs  and 
impacts  involved;  otherwise,  practices  may  not  be  fol¬ 
lowed  and  can  instead  be  harmful  to  a  project. 

Next,  Joe  H.  Findley  brings  us  Measure  Tike  a 
Fighter  Pilot.  Learn  how  measurement  analysts  can  ben¬ 
efit  from  a  focused  strategy  such  as  the  Observe, 

Orient,  Decide,  and  Act  Loop.  In  Applying  Functional 
TSP  to  a  Maintenance  Project,  EUen  George  and  Dr. 

Steve  Janiszewski  share  their  success  in  employing 
both  the  Personal  Software  Process  and  Team 
Software  Process  in  maintenance  project  environ¬ 
ments.  Finally,  David  P.  Quinn  discusses  why  organi¬ 
zations  need  to  be  careful  when  using  project  meas¬ 
ures  as  a  means  of  rewarding  employee  good  per¬ 
formance  in  Tying  Project  Measures  to  Performance 
Incentives. 

As  software  acquisition,  development,  and  sustain¬ 
ment  in  the  Department  of  Defense  continues  to 
challenge  us,  I  hope  this  month’s  set  of  articles  pro¬ 
vides  helpful  information  as  your  team  strives  to  suc¬ 
ceed  and  meet  customer  needs. 

Tracy  L.  Stauder 
Publisher 


Top  5  Department  of  Defense 
Program  Awards  2004 

The  Office  of  the  Under  Secretary 
of  Defense  and  the  National 
Defense  Industrial  Association  pres¬ 
ent  the  Top  5  Department  of 
Defense  Program  Awards  2004. 
These  programs  were  chosen  for 
their  excellence  and  success  at  using 
well-defined  and  proven  processes  to 
develop,  manage,  and  integrate  soft¬ 
ware  into  deliverable  systems.  The 
programs  are  listed  alphabetically 
and  do  not  indicate  a  place  order. 

•  Lightweight  Handheld  Mortar 
BaUistic  Computer 

Service;  U.S.  Army 
Industry  Contractor:  None 

•  Marine  Corps  Total  Force 
System 

Agency:  Technology  Services 
Organization,  Defense  Finance  and 
Accounting  Service 
Industry  Contractor:  Computer 
Sciences  Corporation 

•  Near  Imaging  Field  Tower 
Implementation 

Agency:  Naval  Research  Laboratory 
Midway  Research  Center 
Industry  Contractor:  Joint  effort 
between  Mnemonics,  Inc., 

Assurance  Technology  Corp., 

Harris  Corp.,  Blaseware,  Analex, 
and  SAIC 

•  SmartCamSD  System 

Agency:  NASA 

Industry  Contractor:  Rapid  Imaging 
Software 

•  Warfighter’s  Simulation 

Service:  Program  Executive  Office- 
Simulation,  Training,  and 
Instrumentation,  U.S.  Army 
Industry  Contractor:  Lockheed 
Martin  Simulation  Training  and 
Support 
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Department  of  Defense  Program  Awards 


Lightweight  Handheld 
Mortar  Ballistic  Computer 


Mike  Patriarca 
Project  Manager  Mortars 


Mark  Zhelesnik 

Armament  Research  Development  and  Engineering  Center 


While  supporting  Operation  Iraqi  Freedom,  the  enhanced  functionality  of  the  Eightweight  Elandheld  Mortar  ballistic 
Computer  (LEIMBC)  in  providing  updated  ballistic  solutions  to  fire  missions  for  infantry  mortar  combat  units  has  proven 
to  be  a  great  success.  With  the  EELMBC,  mortar  fire  direction  centers  are  computingfaster,  more  accurate  ballistic  solutions 
to  fire  missions  for  60  mm,  SI  mm  and  1 20mm  mortar  systems. 


Today’s  fast-paced  batdefield  requires 
responsive  and  accurate  weapon  plat¬ 
forms  to  stay  ahead  of  enemy  forces.  The 
XM  32  Lightweight  Handheld  Mortar 
Ballistic  Computer  (LHMBC)  incorpo¬ 
rates  advances  in  digital  electronics  and 
software  capabilities  that  make  it  a  superi¬ 
or  fire  control  system.  Jointly,  the  product 
manager  for  Mortar  Systems  (PM 
Mortars)  and  the  U.S.  Army  Research  and 
Development  Center  (ARDEC)  located  at 
Picatinny,  N.J.,  have  developed  and  are 
fielding  the  LHMBC. 

The  LHMBC  provides  the  soldier,  for 
the  first  time,  a  lightweight  handheld  fire 
control  system  with  an  integrated  Global 
Positioning  System  (GPS)  to  determine 
his/her  location,  and  a  modem  that  allows 
for  digital  communication  within  the  fire 
support  network.  The  system  calculates 
balUsfic  solutions  to  fire  missions  and  pro¬ 
vides  fire  support  coordination  measures 
with  functionality,  including  digital  mis¬ 
sions  (Grid,  Polar,  Shift,  Adjust  Fire,  Fire 
For  Effect,  Immediate  Suppression, 
Immediate  Smoke,  Precision  Registration) 
with  methods  of  control  (At  My  Com¬ 
mand,  When  Ready),  six  simultaneous 
missions,  three  final  protective  fires,  prior¬ 
ity  targets,  fratricide  checks,  check  fire, 
single  safety  fan  with  multiple  segments, 
digital  meteorological  data,  and  digital 
variable  message  format  messages  to/ 
from  the  Advanced  Field  Artillery  Tactical 
Data  System  and  the  Forward  Observer 
System. 

This  software-intensive  LHMBC  pro¬ 
gram  makes  use  of  corporate  software 
and  system  engineering  methodologies 
and  processes  that  have  led  to  the  suc¬ 
cessful  accomplishment  of  several  key 
milestones,  including  urgent  materiel 
release  and  fielding  of  36  LHMBC  sys¬ 
tems.  The  Urgent  Materiel  Release  of 

®  Capability  Maturity  Model  and  CMMI  are  registered  in 
the  U.S.  Patent  and  Trademark  Office  by  Carnegie 
Mellon  University. 


LHMBC  Vers.  1.1  to  3/2  StrjTer  Brigade 
Combat  Team  was  approved  in  August 
2004  to  satisfy  an  urgent  U.S.  Army 
requirement  in  support  of  Operation 
Iraqi  Freedom. 

The  LHMBC  urgent  fielding  effort 
was  particularly  unique.  To  meet  accelerat¬ 
ed  fielding,  PM  Mortars  used  ARDEC  as 

**The  LHMBC  provides 
the  soldier,  for  the 
first  time,  a 
lightweight  handheld 
fire  control 
system  with  an 
integrated  Global 
Positioning  System 
(GPS)  to  determine 
his/her  location,  and 
a  modem  that  allows 
for  digital  communication 
within  the  fire 
support  network/* 

a  central  location  to  load  software  onto  a 
host  computer,  assemble  all  hardware 
components  into  kits,  and  stage  all  fire 
control  equipment  for  delivery.  The  pro¬ 
gram  strategy  calls  for  fielding  incremen¬ 
tal  functionality  improvements  with  two 
additional  releases  scheduled  for  fielding 
in  the  third  quarter  of  2005  and  fiscal  year 
2006.  Incremental  development  continues 


through  fiscal  year  2006  with  planned 
fielding  through  fiscal  year  2008. 

This  incremental  fielding  strategy  best 
meets  the  user’s  urgent  requirement.  The 
Incremental  Development  Life-Cycle 
Model  dovetails  with  this  strategy,  thus  it 
was  the  model  of  choice  for  the  LHMBC 
software  development  team.  During  the 
planning  phase  functionality  was  priori¬ 
tized  by  the  user.  This  input  was  used  by 
the  PM  to  determine  the  functionality  of 
each  software  version. 

LHMBC  Development 

LHMBC  is  one  of  several  spin-offs  of  the 
Mortar  Fire  Control  System  (MFCS),  also 
developed  in  a  joint  effort  between  PM 
Mortars  and  ARDEC.  Initially  fielded  in 
fiscal  year  2003,  the  MFCS  program  was 
selected  for  an  Army’s  research  and  devel¬ 
opment  award  in  2003;  its  architecture  is 
covered  by  a  government  patent  (pend¬ 
ing).  Leveraging  the  flexible  MFCS  archi¬ 
tecture  and  reusing  approximately  80,000 
lines  of  MFCS  code  (40  percent  of  the 
total  LHMBC  code)  resulted  in  an  esti¬ 
mated  cost  savings  of  $2.4  million  and  a 
schedule  savings  of  18  months  for  the 
LHMBC  program.  The  system  also  uses 
the  Mortar  Ballistic  Kernel  that  is  com¬ 
mon  to  both  MFCS  and  LHMBC  (an 
additional  40,000  lines  of  reused  code). 

The  LHMBC  software  application 
program  is  hosted  on  a  Ruggedized 
Personal  Digital  Assistant  (R-PDA)  that 
replaces  the  aging  M23  Mortar  Ballistic 
Computer.  The  R-PDA  hardware  is  basi¬ 
cally  a  commercial  off-the-shelf  device 
acquired  through  the  PM  Common 
Hardware  and  Software  (CHS)  located  at 
Ft.  Monmouth,  N.J.  Choosing  available 
hardware  saved  development  time,  main¬ 
tained  commonality  of  fielded  hardware, 
and  reduced  cost  by  utilizing  economies 
of  scale  in  quantities  purchased  through 
PM  CHS.  The  hardware  is  produced  by 
Tallahassee  Technology  Incorporation  of 
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Lightweight  Handheld  Mortar  Ballistic  Computer 


The  U.S.  Armj’s  XM  32  Ughtmeight  Handheld  Mortar  ballistic  Computer  is  a  significant  improve¬ 
ment  over  the  M2 3  Mortar  ballistic  Computer  —  its  predecessor  —  because  it  is  lighter,  provides  digital 
connectivity  to  the  fiire  support  network,  processes  ballistic  solutions  faster,  and  provides  system  location 
through  an  integrated  Global  Positioning  System. 


Florida  and  provided  through  the  PM 
CHS  contract. 

The  LHMBC  incorporates  an  innova¬ 
tive  fire  control  software  application  that 
provides  enhanced  capability  to  the  mor¬ 
tar  operator  by  reducing  response  times; 
providing  enhanced  accuracy;  and  allow¬ 
ing  the  mortar  to  be  more  mobile,  respon¬ 
sive,  and  lethal.  This  advanced  fire  control 
system  provides  effective  means  to  rapid¬ 
ly  disseminate  relevant  data  among 
weapon  and  command-and-control  sys¬ 
tems.  The  LHMBC  provides  unique  capa¬ 
bilities  available  for  the  first  time  ever  on 
a  R-PDA,  with  a  significant  weight  reduc¬ 
tion  from  eight  pounds  to  three  pounds  at 
one-eighth  the  size  of  previously  fielded 
lightweight  fire  control  systems. 

The  ability  to  quickly  react  to  an 
urgent  need  and  the  continuing  success  of 
this  program  is  due  to  the  well-defined 
proven  practices  and  processes  used  to 
develop,  manage,  and  integrate  the  soft¬ 
ware  into  a  state-of-the-art  R-PDA  com¬ 
puter.  The  in-house  software  development 
and  system  integration  team  used  tools 
such  as  Lean  Six  Sigma  processes  and  pro¬ 
cedures,  earned  value  management,  and 
the  Software  Engineering  Institute’s  (SEI) 
Capability  Maturity  Model®  Integration 
(CMMI®)  Level  3  software  development 
process  to  enhance  overall  efficiencies  and 
quality  of  the  software. 

The  Build  Team  and  Processes 

The  LHMBC  integrated  product  team 
(IPT)  consisted  of  in-house  members 
along  with  members  from  the  test  and  user 
communities,  including  actual  soldiers  who 
operate  the  system.  The  test  and  user  com¬ 
munities  were  involved  early  during  system 
development  to  address  any  concerns  or 
issues  and  reduce  the  number  of  potential 
changes  and  rework  required. 

The  user  provided  valuable  feedback 
on  screen  layouts,  content,  and  sequenc¬ 
ing  of  messages  to  process  missions.  To 
ensure  the  soldiers  have  a  reliable  prod¬ 
uct,  intensive  qualification  testing  was 
performed  in  the  Systems  Integration 
Lab,  including  formal  qualification  testing 
and  an  initial  operational  test. 
Furthermore,  an  independent  validation 
and  verification  (IV&V)  process  was  also 
employed  prior  to  the  product  baseline. 
During  the  IV&V  process,  physical  and 
functional  configuration  audits  were  done 
on  the  software. 

The  industry-  and  government-spon¬ 
sored  CMMI  process  improvement  model 
provides  a  set  of  best  practices  that 
address  product  quality,  productivity,  per¬ 
formance,  cost,  and  stakeholder  satisfac¬ 
tion.  CMMI  Level  3  compliance  ensures 


that  all  CMMI-defined  process  areas  at 
Levels  2  and  3  are  implemented.  CAIMI 
Level  3  processes  are  addressed  in  the  fol¬ 
lowing  areas:  requirements  development, 
technical  solutions,  product  integration, 
verification,  validation,  integrated  project 
management,  risk  management,  decision 
analysis  and  resolution,  and  integrated 
supplier  management.  The  ARDEC 
effectively  implemented  these  process 
improvement  efforts  in  the  development 
of  the  LHMBC. 

The  ARDEC  Armament  Software 
Engineering  Center  (SEC)  has  experi¬ 
enced  success  in  implementing  CAIMI 
Level  3-compliant  processes  for  software- 
intensive  system  projects  such  as  the 
LHMBC.  This  success  has  led  to  the  cur¬ 
rent  process  improvement  initiative  that 
resulted  in  upgrading  the  ARDEC 
Armament  SEC’s  Organizational  Stan¬ 
dard  Process  to  CAIMI  Level  5  compli¬ 
ance.  The  LHMBC  and  other  current 
software -intensive  system  developments 
undertaken  at  ARDEC  Armament  SEC 
now  implement  the  CMMI  Level  5-com- 
pliant  Organizational  Standard  Process. 

LHMBC  and  the  other  Level  5-com- 
pliant  development  efforts  are  scheduled 
to  be  appraised  in  the  fall  of  2005  by  SEI 
approved  appraisers.  Software  develop¬ 
ment  and  process  tools  such  as 
Embedded  Visual  Studio  for  C++,  Visual 
SourceSafe,  and  processMax  were  utilized 
for  the  LHMBC  software  development 
process.  These  tools  assisted  the  develop¬ 
ers  and  the  IPT  members  with  data  man¬ 


agement,  configuration  management,  and 
process  management. 

The  LHMBC  is  a  significant  improve¬ 
ment  over  the  M23  Mortar  Ballistic 
Computer  because  it  is  lighter,  provides 
digital  connectivity  to  the  fire  support  net¬ 
work,  processes  ballistic  solutions  faster, 
and  provides  system  location  through 
GPS.  The  combined  efforts  of  the  PM 
Mortars  and  ARDEC  teams,  along  with 
support  from  our  test  and  user  communi¬ 
ty,  has  led  to  the  successful  development 
of  the  infantry’s  first  fully  digitized  light¬ 
weight  mortar  fire  control  system,  thus 
making  the  infantry  mortarman  an  inte¬ 
gral  part  of  the  digitized  battlefield. ♦ 
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U.S.  Marines  -  First  Into  Battle  and  First 
With  a  Unique  Pay  and  Personnel  System 


Jimmy  W  Selph 
Computer  Sciences  Corporation 

The  Marine  Corps  Total  Force  System  (MCTFS)  is  the  only  integrated  military  pay  and  personnel  system  in  the 
Department  of  Defense  (DoD).  MCTFS  manages  more  than  498,000  Marine  records  for  active,  reserve,  and  retired  mem¬ 
bers;  processes  in  excess  of  17  million  transactions  yearly;  and  computes  an  average  gross  payroll  of  $238  million  per  semi¬ 
monthly  pay  period  totaling  $5.7 12  billion  in  payments  annually.  MCTFS  provides  added  value  to  the  customer’s  mission 
—  paying  service  members  accurately  and  on  time.  Cumulative  for fiscal  year  2004,  MCTFS  demonstrated  that  it  paid  all 
active  duty  and  reserve  Marines  on  time  with  99.92  percent  and  99.83  percent  pay  accuracy,  respectively.  To  date  for fiscal 
year  2005,  the  accuracy  is  1 00  percent for  both  active  duty  and  reserve  Marines.  The  positive  emotional  and financial  impact 
of  paying  deployed  Marines  accurately  and  on  time  is  especially  noteworthy  to  the  commanders  in  the  field  in  terms  of  unit 
morale  and  welfare. 


The  U.S.  Marine  Corps  (USMC)  is  one 
tough  customer  with  demanding 
expectations,  especially  when  it  relates  to 
the  Marine  Corps  Total  Force  System 
(MCTFS),  the  USMC’s  integrated  military 
pay  and  personnel  system.  MCTFS  man¬ 
ages  more  than  498,000  Marine  records 
for  active,  reserve,  and  retired  members; 
processes  in  excess  of  17  million  transac¬ 
tions  yearly;  and  computes  an  average 
gross  payroll  of  $238  million  per  semi¬ 
monthly  pay  period,  totaling  $5,712  billion 
in  payments  annually.  Most  importantly, 
MCTFS  needs  to  pay  service  members 
accurately  and  on  time. 

The  Kansas  City  Technology  Services 
Organization  (TSO-KC)  and  their  prime 
contractor.  Computer  Sciences  Corpora¬ 
tion,  are  dedicated  to  ensuring  continuous, 
responsive,  and  effective  military  pay  and 
personnel  information  technology  support. 
The  technical  managers  of  MCTFS  clearly 
understood  the  task  at  hand  when  the  com¬ 
mandant  of  the  Marine  Corps  stated: 

We  remain  committed  to  trans¬ 
forming  our  manpower  processes 
by  leveraging  the  unique  capabili¬ 
ties  resident  in  the  Marine  Corps 
Total  Force  System,  our  fully  inte¬ 
grated  personnel,  pay,  and  man¬ 
power  system  that  serves  active, 
reserve,  and  retired  members  ... 
Additionally,  MCTFS  facilitates  our 
single  source  of  manpower  data, 
directly  feeding  our  Operational 
Data  Store  Enterprise  and  Total 
Force  Data  Warehouse.  This  dis¬ 
tinctive  capability  allows  us  to 
accurately  forecast  manpower 
trends  and  fuels  our  Manpower 
Performance  Indicators,  which 
provide  near  real-time  graphical 


representation  of  the  Corps’  man¬ 
power  status  such  as  our  deploy¬ 
ment  tempo.  [1] 

The  Unique  Pay  and 
Personnel  System 

MCTFS  is  jointly  sponsored  and  funded  by 
the  Defense  Finance  and  Accounting 
Service  (DFAS)  and  the  USMC.  It  is  the 
only  integrated  military  pay  and  personnel 
system  in  the  Department  of  Defense 
(DoD).  System  integration  is  a  major  contributing 
factor  to  successful  management  and  admin¬ 
istration  of  pay  and  personnel  functions 
for  the  DFAS  and  the  USMC.  Having  man¬ 
agement  fom  these  two  organisations  co-located 
with  the  system  developer  (TSO-KC)  has  proven 
to  be  a  hey  component  of  MCTFS  success. 

System  and  business  integration  of 
these  two  domains  (pay  and  personnel) 
means  that  from  a  single  source,  one  event 
can  be  reported  and  MCTFS  will  properly 
update  both  domains.  A  simplified  exam¬ 
ple  of  single  source  reporting  is  when  a 
Marine  is  promoted  in  rank  from  corporal 
to  sergeant.  System  processing  of  the  pro¬ 
motion  transaction  within  MCTFS 
encompasses  all  the  required  program¬ 
ming  logic  to  ensure  both  pay  and  person¬ 
nel  information  are  concurrently  updated 
by  the  single  input  promotion  transaction. 

The  other  components  of  the  armed 
services  utilize  separate  information  sys¬ 
tems  to  manage  pay  and  personnel  func¬ 
tions  and,  therefore,  must  input  multiple, 
and  by  definition  duplicate,  transactions  to 
report  an  event  such  as  a  promotion  in 
rank.  Integration  of  pay  and  personnel 
and  single  source  reporting  is  unique  to 
MCTFS  and  yields  tangible  and  intangible 
benefits.  These  benefits  include  fewer 
resources  to  perform  simplified  input 


reporting  procedures,  seamless  integration 
of  pay  and  personnel  functions,  and  no 
synchronization  problems  between  dis¬ 
parate  pay  and  personnel  systems. 

MCTFS  is  the  single,  authoritative  data 
source  and  system  of  record  for  all  pay 
and  personnel  information  for  Marine 
service  members.  Input  MCTFS  transac¬ 
tions  are  generated  from  a  variety  of  net- 
centric  ancillary  applications  that  include 
stand-alone,  client/ server,  and  Web-based 
systems.  The  stand-alone  systems  such  as 
Remote  Access  Pay  Transaction  Reporting 
System  allow  users  to  create  transactions 
while  not  connected  to  a  network  in  a 
remote  deployed  environment.  These 
transactions  can  then  be  submitted  into 
MCTFS  once  connectivity  is  available. 

Client/server  systems  such  as  the  Unit 
Diary/Marine  Integrated  Personnel  Sys¬ 
tem  allow  users  to  work  in  a  Wide  Area 
Network  environment,  share  data,  and 
pool  transactions  with  other  system  users. 

The  award-winning‘  MCTFS  front-end 
Web-based  system  Total  Force  Admin¬ 
istration  System  (TFAS) /Marine  OnLine 
provides  users  the  ability  to  generate  self- 
service  transactions.  Transactions  originat¬ 
ing  from  these  ancillary  systems  are 
processed  by  MCTFS  in  conjunction  with 
other  transactions,  and  are  subsequently 
available  for  querying  by  users  in  the 
Operational  Data  Store  enterprise  (ODSe), 
an  Oracle  lOg  relational  database.  This 
database  is  optimized  to  support  the  entire 
Marine  Corps  and  the  breadth  of  process 
stakeholders  and  decision  makers. 

Founded  on  a  strict  set  of  business 
rules,  MCTFS  is  architected  on  the  prem¬ 
ise  that  Marine  commanding  officers  are 
accountable  and  responsible  for  the  per¬ 
sonnel  and  pay  management  of  Marines. 
MCTFS  stores  data  that  includes  opera- 
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The  Total  Tone  System  winning  team  included,  from  left,  Director  Clinton  L.  Swett,  Defense  Finance 
and  Accounting  Service,  Technology  Services  Organisation;  Chief  Warrant  Officer  Harry  Sanches 
U.S.  Marine  Corps  Manpower  and  Reserve  Affairs;  Robert  Frown,  U.S.  Marine  Corps  Manpower 
and  Reserve  Affairs;  and  Director  Gary  Hayes,  Kansas  City  Operations,  Computer  Sciences 
Corporation. 


dons  and  deployment  dates,  promotions, 
performance  evaluations,  duty-station 
assignments,  personal  awards,  reserve 
drills,  skills  and  occupations,  and  train¬ 
ing/education  information.  MCTFS  con¬ 
tains  data  to  correctiy  pay  every  Marine 
with  regard  to  state /federal  income  taxes, 
residency  information,  entidements  and 
allowances,  special  incentive  pay,  and  allot¬ 
ments.  Manpower,  personnel,  pay,  and 
training  data  are  readily  available  via  the 
ODSe  component  in  the  form  of  prefor¬ 
matted  reports  or  via  commercial  off-the- 
shelf  software  such  as  Cognos  Impromp¬ 
tu  and  Powerplay  tools. 

Software  Engineering 
Maturity 

Software  quality  assurance  and  organiza¬ 
tional  institutionalization  of  software 
engineering  practices  are  keys  to  ensuring 
timely  and  accurate  Marine  pay  and  per¬ 
sonnel  administration.  MCTFS  achieved 
the  Software  Engineering  Institute’s  (SEI) 
Capability  Maturity  Model®  for  Software 
(SW-CMM®)  Level  2  rating  in  1997,  and 
SW-CMM  Level  3  rating  in  2000.  The  key 
process  areas  associated  with  Level  2  and 
Level  3  are  thoroughly  institutionalized. 
MCTFS  has  institutionalized  processes 
consistent  with  SW-CMM  Level  4,  and 
was  informally  assessed  at  SW-CMM 
Level  4  in  May  2004  by  an  SEI  Authorized 
Lead  Appraiser.  A  formal  SW-CMM  Level 
4  assessment  was  conducted  in  June  2005/ 
even  while  plans  are  afoot  to  transition  to 
CMM  Integration™. 

TSO-KC  subjects  its  family  of  systems 
to  annual  benchmarks  conducted  by 
Gartner,  Inc.  <www.gartner.com>. 
Benchmarking  conducted  by  Garmer,  Inc. 
captures  and  stores  information  technolo¬ 
gy  performance  metrics  then  compares 
those  TSO  metrics  to  metrics  obtained 
from  similar  participating  organizations. 

Since  1997,  the  average  defect  rate  per 
1,000  function  points  (FPs)  for  MCTFS  is 
1.76.  This  low  average  defect  rate  consis- 
tendy  ranks  MCTFS  in  the  top  quartile  of 
its  software  industry  database.  The  quality 
of  the  MCTFS  software  releases  contin¬ 
ues  to  exceed  expectations  based  on 
empirical  data. 

TSO-KC  managers  attribute  much  of 
the  success  to  the  institutionalized  soft¬ 
ware  process  improvement  activities  that 
ensure  managerial  and  engineering  func¬ 
tions  are  standardized,  defined,  and  docu¬ 
mented  throughout  the  life  cycle.  The  fol¬ 
lowing  highlights  are  from  the  most  cur- 

CMM  Integration  is  a  service  mark  of  Carnegie  Mellon 

University. 

®  CMM  is  registered  in  the  U.S.  Patent  and  Trademark 

Office  by  Carnegie  Mellon  University. 


rent  Gartner  benchmark  for  MCTFS: 

•  FPs  supported  per  person  at  4,623  is 
89  percent  better  than  government 
peer  group  (2,440  FP/full-time 
employees) . 

•  Cost  per  FP  supported  at  $33  is  8  per¬ 
cent  lower  than  the  government  peer 
group. 

•  Cost  per  FP  developed  at  $168  is  54 
percent  lower  than  the  database  aver¬ 
age  and  81  percent  lower  than  the  gov¬ 
ernment  peer  group. 

•  The  903  FPs  developed  per  person  is 
five  times  higher  than  the  government 
peer  group. 

•  Quality  levels,  as  measured  by  defects 
per  1,000  FPs  (slightly  more  than  five, 
including  cosmetic  defects),  remain 
lower  (better)  than  average  for  support. 

•  The  complexity  of  the  business  envi¬ 
ronment  in  development  is  higher  than 
the  government  peer  average  for  both 
enhancements  and  new  development. 
The  USMC,  TSO-KC,  and  their  prime 

contractor  Computer  Sciences  Corpo¬ 
ration  are  proud  of  their  tradition  of 
cooperation  in  managing  MCTFS  and  its 
family  of  integrated  applications.  MCTFS 
today  represents  the  government’s  and 
industry’s  leading  practices,  hardware, 
software,  and  programming  languages. 
MCTFS  will  meet  the  Marine  Corps’ 
needs  into  the  next  decade  and  beyond: 
the  system  is  sufficientiy  scalable  and 
adaptable  to  provide  capabilities  for  enter¬ 


prise-wide  use  within  the  DoD.^ 

Reference 

1 .  Hagee,  Gen.  Michael  W  Senate  Armed 
Services  Committee,  10  Feb.  2005. 

Note 

1.  TFAS  was  awarded  the  2003  Depart¬ 
ment  of  Defense  Chief  Information 
Officer  Award,  the  2003  Department 
of  the  Navy  e-Government  Award,  and 
the  2003  Government  Computer  News 
Innovation  in  Technology  Award. 

2.  The  assessment  resulted  in  MCTFS 
achieving  SW-CMM  Level  4. 
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A  NIFTI  Solution  to 
Far-Field  Antenna  Transformations 


Windie  Borodin  and  Danielle  King 
Midway  Research  Center 

The  Near  Imaging  Field  Tower  Implementation  was  developed  to  provide  a  mid-range  system  that  performs  far-field  charac¬ 
terisation  of  several  60-foot  antenna  systems.  The  challenge  was  to  tah  a  highly  complex  development  effort  and  bring  it  in 
on  time  and  within  cost.  This  article  describes  the  team’s  success  using  an  object-oriented  design  under  a  tailored  IBM  Rational 
Unified  Process  and  established  metrics  to  monitor  the  progress. 


The  mission  of  the  Midway  Research 
Center  (MRC)  located  in  Stafford,  Va., 
is  to  function  as  a  high-precision  signal 
source  for  calibrating  and  testing  national 
and  tactical  systems.  Among  its  numerous 
assets  are  three  antennas  60-foot  in  diam¬ 
eter  that  provide  highly  accurate  radio  fre¬ 
quency  (RF)  signals.  Due  to  their  size  and 
location,  they  require  a  system  to  charac¬ 
terize  the  antennas  to  be  located  as  much 
as  25  miles  away  from  where  they  are  actu¬ 
ally  located.  This  type  of  system  requires 
aircraft  to  accomplish  this  task,  making  it 
economically  impossible  to  characterize 
the  antennas  on  a  regular  basis. 

The  U.S.  Navy  decided  to  develop  a 
mid-range  calibration  system  approxi¬ 
mately  884  feet  away  from  the  antennas  to 
transform  the  RF  data  from  the  antenna 
into  a  pattern  that  makes  them  appear  25 
miles  away.  This  is  known  as  the  Near 
Imaging  Field  Tower  Implementation 
(NIFTI)  project.  The  goal  was  to  produce 
an  operational  system  that  could  be  used 
on  a  regular  basis  to  collect  data  from  the 
test  signal  being  transmitted  from  the 
antennas.  By  analyzing  this  data,  the  sys¬ 
tem  would  be  able  to  determine  what  fac¬ 
tors  are  needed  to  correct  the  actual  signal 
being  transmitted.  This  would  produce  a 
better  quality  and  higher  precision  signal 
being  transmitted  to  the  desired  targets. 

The  system  was  developed  and  tested 
using  the  Antenna  Tracking  Subsystem 
(ATS)-3  antenna.  This  antenna  scans 
across  a  reflector  that  is  mounted  at  the 
calibration  tower.  Data  is  collected  as  the 
ATS-3  antenna  generates  a  continuous 
wave  tone  and  scans  across  the  calibration 
tower.  This  data  is  then  transformed  into 
a  far-field  pattern  using  a  Fourier  trans¬ 
form  technique,  a  method  for  analyzing 
periodic  functions. 

NIFTI  Team 

The  core  development  team  was  located 
on  site  with  the  system  user  and  was  com¬ 
prised  of  both  government  and  contrac¬ 
tor  personnel.  Within  the  development. 


tasks  were  distributed  among  numerous 
contractors,  including  Mnemonics,  Inc.; 
Assurance  Technology  Corporation; 
Harris  Corporation;  Blazeware;  Analex; 
and  Science  Applications  International 
Corporation.  The  project  overcame  chal¬ 
lenges  inherent  when  different  companies 
work  side-by-side  such  as  corporate 
alliances  and  company  policies.  It  also 
tackled  the  required  knowledge  of  both 
RF  theory  and  software  engineering. 

Those  chosen  for  the  task  ranged  in 
experience  from  interns  to  Ph.D.s. 
Expertise  in  different  areas  was  shared 
among  team  members.  Junior  level  soft¬ 
ware  engineers  trained  senior  level  person¬ 
nel  on  programming  best  practices  and 
software  engineering  principles,  while  the 
senior  level  personnel  trained  others  on 
RF  theory.  This  coordination  provided  for 
maximum  use  of  resources  and  overall 
team  strength. 

Developing  alongside  the  ultimate  sys¬ 
tem  user  also  aided  project  success.  The 
NIFTI  team  established  processes  (i.e., 
document  and  code  reviews,  risk  manage¬ 
ment,  configuration  management,  and 
training)  that  engaged  the  customer  (U.S. 
Navy)  and  the  end  user  (MRC  operators) 
in  the  design  and  development  process. 
These  processes  required  the  end  user  to 
be  an  active  participant  in  the  develop¬ 
ment  effort.  The  end  user  participated  in 
aU  document  reviews,  user  interface  work¬ 
ing  groups,  all  system  integration,  and 
acceptance  testing. 

The  project  team  established  a  training 
process  with  the  end-user  software  main¬ 
tenance  group  where  the  software  main- 
tainers  became  part  of  the  development 
team  to  learn  and  understand  the  software 
before  it  was  delivered.  The  end  user  also 
held  a  seat  on  the  Configuration  Control 
Board  and  was  able  to  generate  change 
requests  against  the  system  at  any  time 
during  the  development  process.  This 
allowed  the  end  user  and  customer  the 
ability  to  provide  input  into  the  design  and 
determine  what  changes  were  important. 


Before  this  effort  began,  the  end  user 
was  trained  along  with  the  developers  on 
any  new  tool  or  process  that  was  being 
used  during  the  development.  The  team 
collected  metrics  on  process,  performance, 
change  requests,  budget,  and  schedule. 
These  were  provided  at  every  design  review 
to  the  customer  and  end  user.  This  kept  the 
customer  informed  about  the  project’s 
progress.  It  also  established  a  close  working 
relationship,  which  allowed  for  the  end 
user/customer  to  be  involved  in  every 
major  step  of  the  development  process. 

Software  Development 

The  NIFTI  project  was  developed  using 
object-oriented  design  and  a  tailored  IBM 
Rational  Unified  Process.  A  development 
suite  was  established  consisting  of  the 
Rational  Suite  of  software  that  included 
ClearCase,  ClearQuest,  Requisite  Pro, 
Rose,  Test  Manager,  and  McCabe  Quality 
Toolset.  A  combination  of  UNIX  (Sparc 
processors)  and  Windows  (PC) -based 
servers  were  used  to  house  the  develop¬ 
ment  environment. 

The  project  followed  the  inception, 
elaboration,  construction,  and  transition 
phases.  In  the  inception  phase,  the  budget 
and  high-level  schedule  were  established 
and  the  requirements  were  analyzed.  As 
part  of  performing  requirements  analysis, 
all  stakeholders  met  to  come  to  an  under¬ 
standing  of  the  requirements.  Once  this 
was  accomplished,  the  requirements  were 
then  placed  into  Requisite  Pro  (a  require¬ 
ments  tracking  tool).  Each  requirement 
was  assigned  attributes  such  as  (1)  build 
that  the  requirement  was  to  be  imple¬ 
mented,  (2)  asset  for  which  the  require¬ 
ment  was  needed,  (3)  use  case  that  con¬ 
tained  the  requirement,  and  (4)  test  case  it 
was  assigned.  Also  contained  in  the  data¬ 
base  was  an  interpretation  of  the  require¬ 
ment  that  clearly  explained  the  intent, 
which  was  agreed  to  by  all  stakeholders. 

The  end  of  the  inception  phase  was 
designated  by  a  review  to  the  customer 
and  end  user.  At  this  review,  the  team  pre- 
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sented  the  project  overview,  system  con¬ 
cept,  external  boundaries  of  the  system 
being  designed,  software  standards, 
requirements,  acceptance  criteria  and  veri¬ 
fication  matrix,  and  project  risks. 

For  the  elaboration  and  construction 
phases,  an  iteration  plan  was  developed 
along  with  a  schedule  for  that  phase.  The 
iteration  plan  contained  what  the  team 
was  going  to  accomplish  during  that  phase 
along  with  evaluation  and  exit  criteria,  and 
the  requirements  that  were  to  be 
addressed.  At  the  end  of  each  phase,  the 
team  provided  a  review  to  the  customer 
and  end  user.  During  the  elaboration 
phase,  the  team  focused  on  prototyping 
some  of  the  high-risk  areas  in  the  soft¬ 
ware.  The  main  items  presented  during 
the  end-of-phase  review  were  system 
overviews,  prototype  results,  use  cases, 
test  plans,  refined  software  estimates, 
defined  architectures,  requirements  map¬ 
ped  to  use  cases,  requirements  mapped  to 
construction  iterations,  structured  soft¬ 
ware  models,  schedules,  and  risks. 

The  construction  phase  was  divided 
into  three  builds,  with  each  build  being 
four  to  six  months  in  duration.  One  obsta¬ 
cle  the  project  faced  was  that  the  NIFTI 
system  was  being  installed  into  an  opera¬ 
tional  system,  thus  the  system  was  not 
always  available  for  testing  some  require¬ 
ments.  When  this  occurred,  the  require¬ 
ment  was  deferred  until  the  next  build  and 
a  requirement  initially  slated  for  a  later 
build  was  implemented  in  its  place.  During 
the  build,  the  tester  conducted  usability 
tests  with  the  end  user  that  gave  him  the 
opportunity  to  use  the  system.  Feedback 
was  given  to  the  developers  so  that 
required  changes  could  be  made.  This 
helped  produce  a  system  that  was  user- 
friendly  and  met  the  end  user’s  needs. 

The  reviews  at  the  end  of  each  build 
were  assessments  of  how  the  project  per¬ 
formed  during  that  build  and  whether  the 
project  accomplished  the  goals  contained 
within  the  iteration  plan.  It  also  included  a 
presentation  of  the  metrics  gathered  not 
only  during  the  build,  but  also  during  the 
project  life  cycle.  During  the  end-of-build 
presentation,  a  demonstration  of  the  sys¬ 
tem  was  also  given  to  the  customer.  The 
unique  thing  about  the  demo  was  that  it  was 
run  by  the  end  user.  This  demonstrated  to 
the  customer  that  the  end  user  was  part  of 
the  team  and  that  their  needs  were  being 
addressed.  Also  presented  at  the  review 
were  the  plans  for  the  next  build  along  with 
any  updated  schedule  or  project  risks. 

The  transition  phase  consisted  of  a 
Developers’  Test  and  Evaluation  and  for¬ 
mal  installation  of  the  system  into  the 
operational  asset.  The  testing  was  wit¬ 


nessed  and  signed  off  on  by  personnel 
from  the  site’s  Systems  Engineering 
Department;  also  present  were  personnel 
from  the  operations  staff  —  the  eventual 
system  user. 

Project  Monitoring 

The  NIFTI  project  was  monitored  by 
using  these  established  metrics: 

1.  Cost  measurements.  Within  budget. 

2.  Schedule  performance.  Within  an 
18-month  development  cycle. 

3.  Assessment  measurement.  Pro¬ 
duced  12  source  lines  of  code  (SLOC) 
per  day  versus  the  seven  SLOC  indus¬ 
try  standard. 

4.  Open/ closed  convergence  of  de¬ 
fects.  Number  of  open  defects  con¬ 
verged  with  closed  defects. 

5.  Hours  expended  per  defect.  Most 
defects  were  repaired  in  less  than  three 
hours. 

6.  SLOC  per  defect.  Most  defects  were 
repaired  in  less  than  10  lines  of  code. 

7.  Defects  per  subsystem  inspection 
and  testing  (I&T)  versus  system 

I&T.  Most  defects  were  detected  dur¬ 
ing  subsystem  I&T  before  being  deliv¬ 
ered  to  the  testers. 

The  team  also  monitored  the  software 
to  evaluate  it  in  terms  of  complexity  (all 
modules  had  a  complexity  of  10  or  less 
using  McCabe  Tools),  to  check  for  memo¬ 
ry  leaks,  to  determine  bottlenecks,  and  to 
test  path  coverage. 

Summary 

The  innovative  processes  that  incorporat¬ 
ed  the  end  user  and  customer  throughout 
the  software  development  process  have 
proven  to  be  key  in  achieving  project  suc¬ 


cess.  These  strucmred  processes,  risk  mit¬ 
igation,  metrics,  and  close  communication 
with  the  end  user/customer  have  been 
instrumental  in  producing  a  quality  prod¬ 
uct  on  time  and  within  budget  that  met  its 
requirements.  The  diverse  backgrounds  of 
the  development  team  also  proved  to  be 
instrumental  in  the  success  of  the  project. 

Major  benefits  from  the  system  were 
seen  even  before  it  was  officially  turned 
over  to  operations:  It  has  been  used  to 
detect  a  feed  offset  and  bore  sight  prob¬ 
lem  in  the  ATS’,  and  to  find  a  software 
problem  that  caused  antenna  tracking  to 
be  off  in  customer  tests.  The  clients  were 
grateful  that  these  and  other  problems 
could  now  be  identified  and  resolved 
much  faster  and  easier  than  before.^ 

Project  Points  of  Contact 

Windie  Borodin 
Midway  Research  Center 
Naval  Research  Laboratory 
635  Telegraph  RD 
Stafford, VA  22554 
Phone:  (703)  551-1992 
Fax:  (703)  221-3317 
E-mail:  borodin@kingcrab. 
nrl.navy.mil 

Danielle  King 

Midway  Research  Center 

Naval  Research  Laboratory 

635  Telegraph  RD 

Stafford, VA  22554 

Phone  (703)  551-1975 

Fax:  (703)  221-3317 

E-mail:  king@kingcrab.nrl.navy.mil 


September  2005 


www.stsc.hill.af.mil  9 


Department  of  Defense  Program  Awards 


SmartCam3D  Provides  New  Levels 
of  Situation  Awareness 
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SmartCam3D  (SC3D)  is  a  hybrid  synthetic  vision  system  that  combines  live  sensor  information  with  data  from  a  synthetic  vision 
system  to  create  a  virtual  cockpit  window.  This  combination  of  technologies  allows  the  system  to  circumvent  some  of  the  limita¬ 
tions  from  each  approach.  SC3D  is  supporting  various  Department  of  Defense,  NASA,  and  commercial  industry  endeavors 
related  to  the  operation  of  tele-operated  and  windowless  cockpit  designed  vehicles. 


The  pertinent  information  related  to 
the  environment  and  operational 
envelope  of  a  vehicle  is  commonly  re¬ 
ferred  to  as  situation  awareness  (SA).  A 
lack  of  SA  is  responsible  for  numerous 
aircraft  accidents  that  claim  hundreds  of 
fatalities  per  year.  Additionally,  the  mone¬ 
tary  cost  associated  with  poor  SA  is  in  the 
billions  of  dollars  due  to  lost  aircraft, 
increased  training  time,  decreased  capabil¬ 
ities,  and  operational  inefficiencies.  The 
Department  of  Defense  (DoD),  NASA, 
and  the  general  aviation  industry  have  aU 
lost  important  assets  because  an  operator 
did  not  have  adequate  SA. 

The  development  of  a  visualization  sys¬ 
tem  that  would  greatly  improve  a  pilot’s  SA 
would  prove  very  valuable  to  both  govern¬ 
ment  and  commercial  organizations. 
SmartCamSD  (SC3D)  was  developed  to 
serve  this  function  and  has  provided  SA  to 
pilots  at  levels  not  previously  possible. 

Development  of  SC3D  principles  and 
technology  components  began  in  1996  as 
parallel  efforts  by  Rapid  Imaging  Software 
(RIS)  and  NASA.  RIS  was  developing  a 
commercial  product  that  would  allow  for 
the  visualization  of  terrain.  At  the  same 
time,  NASA’s  Johnson  Space  Center  was 
developing  visualization  technologies  that 
would  be  used  on  the  X-38  program.  The 
X-38  program  was  a  technology  develop¬ 
ment  program  that  was  building  a  series  of 
flight-test  vehicles  to  determine  what  would 
be  required  to  develop  a  fully  functional 
crew  return  vehicle  for  the  International 
Space  Station.  In  1997  both  organizations 
developed  a  collaborative  parmership  to 
develop  an  advanced  visualization  system 
that  would  greatly  enhance  a  pilot’s  SA. 

A  hybrid  synthetic  vision  system  (HSVS) 
combines  live  sensor  data  with  information 
from  a  synthetic  vision  system  (SVS)  to  cre¬ 
ate  real-time,  information-rich  visuals.  An 
SVS  provides  a  computer  representation  of 
the  operational  environment  and  is  typically 
created  using  digital  elevation  data,  imagery, 
maps,  charts,  vehicle  models,  etc. 


SVS  visuals  are  typically  full  of  informa¬ 
tion  and  can  be  used  during  day,  night,  or 
low-visibility  conditions.  An  SVS  by  itself 
can  suffer  from  data  freshness  problems. 
These  problems  can  occur  because  the  SVS 
visuals  are  created  from  data  typically  col¬ 
lected  by  a  satellite  or  aircraft  flying  over¬ 
head,  meaning  the  data  could  have  been  col¬ 
lected  days,  weeks,  months,  or  years  ago.  In 
a  dynamic  environment,  the  information 
could  be  outdated  and  provide  obsolete  or 
inaccurate  scene  information. 

A  live  sensor  system  can  provide  up-to- 
the-second  information  for  the  region  of 
interest,  but  by  itself  is  not  very  useful  when 
visibility  conditions  are  hampered  by  rain, 
snow,  sand,  fog,  and  smoke,  which  are  fre¬ 
quently  encountered  by  vehicles. 

HSVS  visuals  allow  the  user  to  circum¬ 
vent  some  of  the  limitations  from  each  sep¬ 
arate  approach  by  providing  real-time  infor¬ 
mation-rich  visuals.  SC3D  has  become  the 
government’s  premier  HSVS  system.  The 
SC3D  system,  with  its  innovative  concepts 
and  one-of-a-kind  software  technology, 
blurs  the  traditional  distinction  between 
visual  and  instrumented  flights/ operations. 

Strategies  and  Methodologies 

The  team  organization  played  an  important 
factor  in  the  overall  success  of  the  SC3D. 
This  team  included  experts  with  hands-on 
experience  in  software  development,  human 
factors,  visualization,  configuration  manage¬ 
ment,  avionics,  and  aviation  piloting.  Other 
major  factors  that  played  a  significant  role  in 
the  success  of  the  SC3D  system  were  the 
processes  that  were  followed  and  the  design 
decisions  made  during  the  planning  phase. 

The  system  was  developed  using  a  spiral 
development  approach  implementing  agile 
programming  practices,  object-oriented  de¬ 
sign  (OOD),  and  object-oriented  program¬ 
ming  (OOP).  The  team  also  followed  a  rig¬ 
orous  quality  assurance  program  through¬ 
out  the  entire  development,  implementa¬ 
tion,  and  deployment  life  cycle.  Spiral  devel¬ 
opment  has  allowed  users  to  see  an  increas¬ 


ingly  more  functional  system  throughout 
the  development  of  the  SC3D  system.  It 
also  allowed  for  an  iterative  approach  to 
requirements  development  by  providing 
successive  spiral  cycles  that  contained  the 
lessons  learned  from  previous  spiral  cycles. 
Agile  programming  permitted  the  developer 
to  incorporate  user  feedback  into  the  soft¬ 
ware  within  each  spiral  cycle.  The  OOD  and 
OOP  principles  simplified  maintenance, 
reuse,  and  capability  augmentation  of  the 
core  technologies. 

Any  problems  encountered  could  be 
quickly  isolated  to  individual  components 
making  problem  identification  and  resolu¬ 
tion  much  easier.  As  new  capabilities  were 
needed,  the  various  components  were  easily 
created/modified.  Although  the  SC3D  sys¬ 
tem  has  more  features  than  any  visualization 
system  currently  available  and  supports  real¬ 
time  visualization  of  very  complex  environ¬ 
ments,  reliability  has  always  been  the  most 
important  factor.  An  intense  quality  assur¬ 
ance  process  was  utilized  that  allowed  for 
the  development  of  a  product  that  users 
could  trust  to  work.  New  algorithms  were 
verified  theoretically  and  experimentally 
before  being  added  to  the  source  code.  Beta 
test  versions  of  the  algorithms  were  exten¬ 
sively  tested  by  various  organizations  com¬ 
posed  of  users  from  NASA,  the  DoD,  and 
industry.  Bug  reporting,  tracking,  and  reso¬ 
lutions  were  included  in  the  software  quality 
assurance  process. 

SC3D,  which  is  comprised  of  approxi¬ 
mately  64,000  lines  of  code,  has  met  the 
required  specifications  regarding  perform¬ 
ance,  functionality,  and  reliability.  The  initial 
performance  requirement  called  for  refresh 
rates  between  five  hertz  and  10  hertz. 
Testing  indicates  that  the  SC3D  system  pro¬ 
vides  much  better  performance  than  this, 
with  refresh  rates  between  10  hertz  and  30 
hertz.  The  system  functionality  requirement 
called  for  enhanced  SA  to  be  provided  to 
operators  of  remotely  piloted  vehicles.  The 
successful  integration  and  use  by  multiple 
military  branches,  various  NASA  programs. 
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and  many  other  government,  commercial, 
and  education  organi2ations  is  proof  that 
the  system  can  provide  windowless  cockpit 
and  remote  ground  station  (tele-operation) 
visuali2ation.  The  embracement  of  the 
SC3D  system  by  many  organi2ations,  where 
Uves  depend  on  the  reliability  of  the  infor¬ 
mation,  is  a  testament  to  its  excellence. 

Weekly  and  quarterly  reports  were  used 
as  part  of  an  Earned  Value  Management 
process  to  track  schedule,  budget,  and 
deliveries.  Weekly  reports  highlighted  the 
technical  accomplishment  for  the  week 
and  outlined  what  would  be  done  the  fol¬ 
lowing  week.  Quarterly  reports  were  com¬ 
prehensive  and  included  the  money  allo¬ 
cated  for  the  quarter,  percentage  of  physi¬ 
cal  work  completed  to  date,  and  the  work 
that  would  be  performed  the  upcoming 
quarter.  The  average  Cost  Performance 
Index  and  average  Schedule  Performance 
Index  for  the  third  SC3D  spiral  develop¬ 
ment  cycle  of  the  project  were  0.99  and 
I.O  respectively.  The  SC3D  project  met  or 
exceeded  aU  scheduled  software,  hardware, 
and  documentation  deliveries. 

User  Base 

A  testament  to  the  enormous  capabilities, 
quality,  and  reliability  provided  by  the  SC3D 
system  is  the  large  number  and  diversity  of 
users  that  have  enjoyed  the  benefits  provid¬ 
ed  by  the  SC3D  system.  SC3D  usage  con¬ 
sists  of  individuals  from  more  than  90  dif¬ 
ferent  organi2ations,  including  the  follow¬ 
ing:  DoD,  NASA,  education  institutions,  re¬ 
search  institutions,  European  Space  Admin¬ 
istration,  and  many  industrial  organi2ations. 

The  SC3D  system  supports  several 
branches  of  the  military  by  providing  func¬ 
tionality  in  many  major  areas  that  include 
combat/ reconnaissance  visuali2ation  for 
unmanned  aerial  vehicle  (UAV)  operators, 
development  of  new  human  interface  con¬ 
cepts  for  pilots,  training  of  new  and  experi¬ 
enced  pilots,  and  supporting  troops  in  com¬ 
bat.  SC3D  will  also  play  an  important  role  as 
a  recruitment  tool  such  as  the  following: 

•  The  Advanced  Systems  and  Concepts 
Office,  in  the  Assistant  Deputy  Under¬ 
secretary  of  Defense  (ADUSD)  office, 
has  been  investigating  many  uses  of 
SC3D  for  the  warfighter.  Cmdr.  Thomas 
Moore,  special  assistant  in  the  ADUSD 
office  noted: 

SmartCam3D  is  truly  a  one-of-a-kind 
breakthrough  technology  in  situation 
awareness.  There  is  no  other  software 
currently  available  that  provides  these 
benefits  to  the  warfighter.  [1] 

•  The  Air  Force  Research  Lab  (AFRL)  is 
using  SC3D  technology  to  develop  new 


interface  paradigms  for  UAV  operators. 
The  AFRL  Interactive  Visual  Interface 
Environment  for  UAVs  program  has 
integrated  the  SC3D  system  with  a 
Predator  simulator  to  create  an  excellent 
environment  to  test  new  human  inter¬ 
face  concepts  related  to  the  tele-opera¬ 
tion  of  Predator  UAVs.  As  Dr.  Mark 
Draper  (Human  Effectiveness  Direc¬ 
torate  -  AFRL)  noted: 

Within  the  past  year,  this  effort  has 
been  successfully  demonstrated  and 
briefed  to  the  commanders  of 
AFMC,  AFRL,  ASC/RA  and  ACC/ 

DR  UAV  SMO  among  others  . . . 

The  Air  Force  clearly  sees  the  poten¬ 
tial  that  SmartCam3D  technology 
provides.  [2] 

•  The  SC3D  system  has  provided  com¬ 
manders/soldiers  with  enhanced  battle 
SA.  Operators  with  the  I -1 4  Cavalry  in 
Iraq  noted: 

Our  current  mission  is  Force/Per¬ 
imeter  Protection  since  our  immedi¬ 
ate  threat  is  very  real  and  very  close. 
Every  OP  [Observation  Post]  that 
hears  or  sees  something  we  respond 
to.  Having  the  SmartCam-3D  over¬ 
lays  assisted  in  our  response  time.  [3] 

•  The  SC3D  system  will  become  an 
extremely  important  recruiting  tool.  As  a 
project  office  representative  with  the  1st 
Stryker  Brigade  Combat  Team  noted: 

SmartCam3D  is  a  big  hit  with  the 
soldiers.  SmartCam3D  could  be¬ 
come  the  Army's  biggest  recruiting 
tool  to  encourage  young  video 
gamers  to  enlist  and  re-enhst.  [4] 

Another  major  user  of  SC3D  is  NASA, 
which  is  using  the  system  to  support 
endeavors  related  to  flight/aviation  safety, 
visualUation  requirements  definition  for 
future  spacecraft,  mission  planning,  flight 
safety/operations,  and  visualUation  for 
future  Mars  and  Lunar  rovers.  SC3D  tech¬ 
nology,  which  includes  VisualFlight  and 
Landform,  is  also  supporting  various 
endeavors  related  to  basic  research,  accident 
investigations,  educational  outreach,  enter¬ 
tainment,  and  other  efforts. 

Summary 

SC3D  is  the  first  real-time  HSVS  system.  Its 
component-based  architecture  allows  it  to 
be  easily  incorporated  into  other  software 
applications.  The  processes,  technologies, 
and  early  inclusion  of  user  feedback  played 
a  critical  role  in  the  highly  successful  nature 


of  the  SC3D  system.  Its  numerous  innova¬ 
tions  make  it  easy  to  use  and  provide  numer¬ 
ous  capabilities  not  offered  in  any  other 
visuali2ation  system  currently  available. 
SC3D  technology  reduces  uncertainties, 
minimi2es  costs,  and  improves  safety. 

Its  large  user  base  includes  many  federal 
government  agencies,  various  education  in¬ 
stitutions,  the  European  Space  Administra¬ 
tion,  many  commercial  companies,  and  the 
entertainment  industry.  This  large  user  base 
and  highly  successful  flight  utilUation  are  a 
testament  to  the  usability  and  quality  of  the 
software.  The  improved  SA  that  SC3D  pro¬ 
vides  has  proven  very  valuable  to  US.  troops 
in  combat  operations,  reconnaissance  mis¬ 
sions,  and  is  helping  to  neutralUe  terrorist 
threats.  As  a  project  office  representative 
with  the  I"  Stryker  Brigade  Combat  Team 
noted,  “This  software  [SC3D]  will  save 
lives.”^ 
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The  Wa  fighter’s  Simulation  (WARSIM)  is  the  U.S.  Army’s  next  generation  constructive  simulation  capability.  Revo¬ 
lutionary  in  scope,  IPARSIM  allows  commanders  and  their  staffs  to  train  in  the  contemporary  environment  they  will face  in 
Iraq,  Afghanistan,  and  other  trouble  spots  in  the  world.  WARSIM  provides  additional  needed  realism,  will  decrease  the 
resources’  need  to  train,  and  when  combined  with  the  other  systems  of  the  Army’s  Constructive  TrainingTe  deration  will pro¬ 
vide  all  the  capabilities  required for  a  fully  integrated  live-virtual-constructive  training  environment. 


The  Warfighter’s  Simulation  (WARSIM) 
is  a  computer-based  simulation  that 
will  revolutionize  the  way  the  commander 
and  staff  train  and  conduct  mission 
rehearsal  in  the  contemporary  operating 
environment.  The  program  mission  is  to 
design,  develop,  produce,  and  deploy  a 
training  simulation  system  to  support  U.S. 
Army  commanders  and  their  staffs.  The 
WARSIM  architecture  provides  flexibility 
to  interface  with  other  live,  virtual,  and 
constructive  training  simulations  and  to 
simultaneously  interoperate  with  the  organ¬ 
ic  Command,  Control,  Communications, 
Computer,  and  Intelligence  systems  and 
equipment  of  the  training  audience.  WAR¬ 
SIM,  a  software-intensive  system  with 
more  than  two  million  lines  of  code,  pro¬ 
vides  the  Army’s  next  generation  com- 
mand-and-control  training  environment. 

Recent  studies  recognize  a  military  train¬ 
ing  revolution  that  occurred  when  the 
Combat  Training  Centers  (CTCs)  were  cre¬ 
ated  in  the  1980s  [1].  The  collective  training 
at  the  CTC  gives  U.S.  forces  a  decided 
advantage  over  its  adversaries.  Those  same 
studies  cite  the  need  for  a  second  revolution 
in  training,  one  that  will  expand  the  collec¬ 
tive  training  success  of  the  CTC  to  home 
station  and  deployed  units. 

WARSIM  is  part  of  that  second  training 
revolution:  It  provides  a  realistic  battlefield 
environment  that  more  closely  matches  the 
contemporary  operating  environment 
encountered  today  in  Iraq  and  Afghanistan. 
WARSIM  is  perfectly  suited  to  train  the  geo¬ 
graphically  dispersed  modular  Army; 
Brigade  combat  teams  from  different  geo¬ 
graphic  areas  will  be  expected  to  join  in 
forming  Army  or  Joint  Forces  command 
elements.  This  will  demand  a  distributed 
training  and  mission  rehearsal  capability  that 
WARSIM  brings,  and  with  the  high  opera¬ 
tional  tempo  of  today’s  units,  WARSIM  will 
reduce  overhead  personnel  requirements 
p'picaUy  levied  on  training  units. 

The  U.S.  Army  Program  Executive 
Office  for  Simulation,  Training,  and 


Instrumentation  (PEO  STRI),  the  WAR¬ 
SIM  acquisition  agent,  will  meet  the  Army’s 
training  requirements  with  WARSIM  by 
linking  to  Army  and  joint  training  simula¬ 
tions  in  the  near  term,  and  simultaneously 
developing  increased  simulation  capabilities 
applicable  to  both  Army  and  joint  training. 
WARSIM  has  an  open  architecture  that 
enables  it  to  federate  with  other  existing  or 
future  simulations. 

WARSIM  is  presently  in  the  engineering 
and  manufacturing  development  phase  of 
acquisition.  The  prime  contractor  is 
Lockheed  Martin  Simulation,  Training,  and 
Support  (LM-STS),  which  leads  an  industry 
team.  The  intelligence  subcomponent  of 
WARSIM  was  separately  developed  to 
accommodate  top-secret/ sensitive  com- 
partmented  information  requirements  and 
is  now  completely  integrated  within  WAR¬ 
SIM.  The  intelligence  subcomponent  is  an 
integral  part  of  the  overall  Army  strategy 
for  WARSIM. 

The  initial  version  of  WARSIM  was 
delivered  to  Fort  Leavenworth,  Kan.,  in 
December  2004.  Several  factors  contributed 
to  the  success  of  the  program  and  the  satis¬ 
faction  expressed  by  the  WARSIM  cus¬ 
tomer.  Key  among  these  are  the  application 
of  concurrent  engineering,  co-location  of 
project  stakeholders,  creation  of  an  environ¬ 
ment  where  subcontractors  are  teamed  with 
the  prime  contractor,  dedication  to  quantita¬ 
tive  management  of  program  processes  and 
measures  at  Capability  Maturity  Model® 
Integration  Level  4,  an  innovative  approach 
to  integration  and  testing,  and  careful  man¬ 
agement  of  the  program  cost  and  schedule. 

Concurrent  Engineering 

From  its  inception,  WARSIM  has  been  ded¬ 
icated  to  the  concept  of  concurrent  engi¬ 
neering  where  engineering,  program  man¬ 
agement,  acquisition,  and  user  representa¬ 
tives  interact  throughout  the  development  - 
from  requirements  analysis  through  integra¬ 
tion  and  test  -  as  members  of  integrated 
development  teams  (IDTs).  The  entire 


WARSIM  program  is  organized  using  IDTs. 
Every  WARSIM  IDT  includes  members 
from  PEO  STRI  —  the  materiel  developer  — 
the  National  Simulation  Center,  the  combat 
developer,  and  the  Lockheed  Martin  devel¬ 
opment  team.  This  approach  reduces  pro¬ 
gram  risk  significantly  and  ensures  that 
every  stage  of  development  is  validated  and 
correct  across  the  project’s  stakeholders. 

Co-Location 

PEO  STRI  has  been  co-located  with  WAR¬ 
SIM  developers  from  the  very  beginning  of 
the  program.  Contractor  engineers,  cus¬ 
tomers,  user  representatives,  subject  matter 
experts,  and  program  management  shared 
office  space  at  the  development  site,  which 
allowed  stakeholders  to  attend  all  meetings, 
working  groups,  and  ad-hoc  sessions;  this 
made  all  parties  truly  integrated  members  of 
the  WARSIM  team. 

Subcontractor  Teammates 

A  strong  industry  team  developed  the 
WARSIM  product,  and  includes  Lockheed 
Martin  (prime).  Science  Applications 
International  Corporation,  Dynamics  Re¬ 
search  Corporation,  Northrop  Grumman, 
Veridian  (General  Dynamics),  and  Virtual 
Technology  Corporation.  Each  contractor 
brings  a  high  degree  of  technical  expertise 
and  competence  to  the  program.  Another 
key  to  the  successful  teaming  approach  is 
that  all  subcontractors  are  co-located  at  the 
Lockheed  Martin  facility,  sharing  offices 
with  Lockheed  Martin  engineers.  Respon¬ 
sibility  for  development  of  work  products  is 
shared  across  the  teams,  with  members 
from  each  company  developing  each  com¬ 
puter  software  configuration  item.  All  team¬ 
mates  followed  the  same  Lockheed  Martin 
standard  engineering  processes. 

Quantitative  Management 

WARSIM’s  project  management  tracking  metrics 
were  developed  by  analyzing  Lockheed 
Martin  Simulation,  Training,  and  Support 
organizational  goals,  customer  needs  as  doc- 


I  2  Crosstalk  The  Journal  of  Defense  Software  Engineering 


September  2005 


WARSIM  Enters  the  Scene  in  Army  Training 


WARSIM  kadership  are,  from  kft,  Greg  Manross,  WARSIM  project  director,  project  manager 
Constructive  Simulation  PEO  STRJ;  Debra  Palmer,  vice  president  business  Development  and  Advanced 
Programs,  IM-STS;  Ed  Pajne,  WARSIM  program  manager,  IM-STS;  Col  Kevin  Dietrick,  project 
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umented  in  the  contractual  requirements, 
and  industry  standards.  Using  these  goals, 
the  program  established  quantitative  man¬ 
agement  metrics  to  objectively  monitor 
process  and  product  quality.  These  measure¬ 
ments  allowed  management  to  assess 
WARSIM’s  program  status  and  identify  pro¬ 
gram  risks,  which  then  were  managed  and 
mitigated  in  accordance  with  the  WARSIM 
risk  management  process. 

Quantitative  management  allowed  both 
WARSIM  management  and  the  WARSIM 
customer  to  determine  in  real  time  that  the 
processes  and  product  development  were 
meeting  the  project’s  goals,  or  to  take  cor¬ 
rective  action  as  necessary. 

Integration  and  Test 

Eighteen  months  prior  to  delivery,  the  pro¬ 
gram  transitioned  from  a  standard  develop¬ 
ment  effort  to  one  focused  on  evaluating 
the  system  from  the  user’s  perspective.  To 
accomplish  this,  an  integration  and  test 
operability  team  was  established  to  lead 
these  evaluation  efforts  with  all  other  engi¬ 
neering  functions  operating  in  support. 

The  operability  team  was  tasked  with  test¬ 
ing  the  system  as  it  will  be  used  once  deliv¬ 
ered  to  the  users.  This  team  was  composed 
of  skilled  individuals  with  a  strong  back¬ 
ground  in  Army  training  that  worked  close¬ 
ly  with  the  user’s  representatives  at  the 
National  Simulation  Center  (NSC)  to  devel¬ 
op  thread  tests  of  the  functionality.  A  pri¬ 
mary  focus  of  the  operability  testing  was  to 
evaluate  and  enhance  system  usability. 
During  this  period,  the  operability  team  set 
the  priority  for  changes  and  worked  side  by 
side  with  the  software  developers  to  define 
the  desired  changes. 

Key  to  the  concept  of  operability  testing 
was  the  hosting  of  frequent  Continual  User 
Assessments  in  the  development  environ¬ 
ment.  The  benefit  is  win-win  for  both  users 
and  developers:  Users’  training  time  is  great¬ 
ly  reduced  as  they  learn  the  system  as  it  is 
buHt;  developers  gain  valuable  user  feedback 
and  validation  while  developing  code  for  the 
next  WARSIM  build. 

Another  benefit  of  active  user  participa¬ 
tion  was  an  improved  understanding  of 
requirements.  With  a  better  understanding 
of  the  users’  needs,  software  developers 
would  often  prototype  a  capability  and 
demonstrate  it  to  the  user  -  a  process  that 
turned  out  to  be  extremely  efficient  -  allow¬ 
ing  the  software  team  to  add  more  capabili¬ 
ty  to  the  final  product  in  less  time. 

Three  months  prior  to  delivery,  the 
WARSIM  integration  and  test  team  held  a 
three-week  culminating  integration  event, 
commonly  referred  to  as  the  September  ’04 
Event  (S4E).  Its  purpose  was  to  test  the  sys¬ 
tem  in  its  intended  configuration  and  under 


exercise-Hke  conditions  prior  to  delievery. 
This  event  was  conceived  as  a  process 
improvement  to  anticipate  problems  ob¬ 
served  in  the  user’s  environment  not  experi¬ 
enced  in  the  integration  lab  environment. 

To  prepare  for  the  S4E,  the  program 
buHt  a  new,  classified  facility  that  mirrored 
the  WARSIM  installation  at  the  NSC. 
Twenty-four  experts  from  the  Battle 
Command  Training  Program,  who  were 
previously  unfamiliar  with  WARSIM,  partic¬ 
ipated  in  the  event  to  ensure  the  testing  was 
realistic  and  valid;  the  successful  outcome  of 
the  event  enabled  WARSIM  to  identify  a 
final  set  of  changes  and  quickly  incorporate 
them  in  the  months  prior  to  delivery. 
Because  of  its  thorough  user-oriented  focus 
on  integration,  the  program  is  confident  of 
continued  success  as  it  moves  through  the 
next  phase  of  user  testing. 

Schedule  and  Cost 

WARSIM  has  always  met  or  exceeded  its 
commitments  to  cost,  schedule,  and  per¬ 
formance.  After  termination  of  the  Joint 
Simulation  System  (JSIMS)  program  in 
2002,  WARSIM’s  program  was  re-baselined 
to  provide  a  complete  training  system  to  the 
Army.  The  program  made  major  schedule, 
budget,  and  staffing  adjustments  to  accept 
additional  requirements  from  the  JSIMS 
program.  The  contractor  delivered  a  com¬ 
plete  training  system  while  holding  to  the 
committed  delivery  schedule  within  govern¬ 
ment  funding  constraints.  WARSIM  deliv¬ 
ered  its  software  on  Dec.  17,  2004  and 


exceeded  the  user’s  expectations. 

Conclusion 

WARSIM  is  a  key  enabling  program  for  the 
training  of  the  Army’s  current  and  future 
force  commanders  and  staff  It  is  a  critical 
component  in  the  Army  Constructive 
Training  Federation  (ACTF)  that  will  help  to 
bring  about  a  second  revolution  in  military 
training.  WARSIM  will  fulfill  the  Army 
requirement  for  training  its  forces  in  all 
aspects  of  command  and  control  ACTF 
models  will  provide  full  training  functionali¬ 
ty  for  leader  and  battle  staff  computer- 
based  simulation  training  throughout  the 
Army,  joint,  interagency,  intergovernmental, 
and  multinational  spectra.  WARSIM’s  con¬ 
tributions  to  training  today’s  Army,  and 
tomorrow’s  future  forces,  are  just  starting  to 
be  realized.^ 
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The  Myth  of  the  Best  Practices  Silver  Bullet 
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For  many  years,  there  has  been  much  work  attempting  to  identipi  a  set  of  best  practices  that  software-intensive  projects  could 
apply  to  aid  in  the  acquisition,  production,  or  upgrade  of  software.  Spurred  on  by  the  1987  and  2001  Defense  Science  Board 
findings,  efforts  conducted  by  the  Software  Engineering  Institute  and  the  Software  Program  Managers  Network  have  identi¬ 
fied  and  documented  specific  practices  that  have  had  apparent  success  in  loweringproject  risk,  improving  cost  and  schedule  per¬ 
formance,  and  enhancing  user  satisfaction.  Since  Section  804  of  the  National  Defense  Authorisation  Act  for  Fiscal  Year 
2003  was  enacted  on  Dec.  2,  2002  and  became  law,  there  has  been  much  activif  in  this  area,  particularly  in  the  Department 
of  Defense  and  its  various  services.  This  article  explores  some  of  these  efforts,  looks  at  the  practices  that  have  resulted,  and 
attempts  to  examine  certain  key  relationships  that  must  be  considered  when  applying  them  to  projects. 


Best  practices  are  often  looked  on  as 
the  Holy  Grail  of  process  improve¬ 
ment,  the  silver  bullet  that  will  cure  aU  iUs. 
A  manager  might  reasonably  ask,  “After 
all,  couldn’t  I  expect  the  same  degree  of 
success  if  I  use  the  same  processes  in  the 
management,  engineering,  assurance,  and 
tracking  of  the  project?”  The  answer  is  an 
unqualified  maybe. 

Indeed,  far  from  being  a  silver  bullet, 
there  is  some  evidence  that  the  term  best 
practices  Eriks,  significant  meaning.  In  2001, 
Dr.  Richard  Turner  conducted  a  study  for 
the  Department  of  Defense  (DoD)  [1]  to 
identify  credible  best  practices  that  could 
improve  performance,  predictability,  qual¬ 
ity,  and  operational  effectiveness  while 
lowering  risk,  shortening  schedules,  and 
reducing  development  costs.  As  a  result  of 
this  study.  Turner  concluded  that  because 
the  term  best  practices  is  consistentiy  mis¬ 
used,  it  is  misleading  at  best  and  useless  at 
worst.  It  has  become  a  catchall  phrase  that 
bundles  diverse  ideas  about  practices  and 
frameworks,  and  is  used  by  some  to  legit¬ 
imize  unproven  practices,  tools,  or 
processes. 

Unproven  practices,  while  not  desig¬ 
nated  as  best,  are  often  essential  compo¬ 
nents  of  successful  projects.  They  simply 
do  not  have  a  pedigree  outside  of  the 
project  to  which  they  are  being  applied. 

Practice  Relationships 

Software  managers  must  not  assume  that 
just  because  a  certain  practice  has  been 
labeled  best  that  it  will  indeed  improve  the 
performance,  predictability,  quality,  and 
operational  effectiveness  of  the  software 
they  are  responsible  for  producing.  Nor 
does  it  mean  they  should  view  these  prac¬ 
tices  with  outright  suspicion,  but  only  that 
they  must  understand  the  advantages  and 
disadvantages  of  the  practices  for  their 
particular  projects  and  how  they  can  be 


usefully  adapted  to  the  various  needs  of 
their  organization. 

To  understand  why  so-called  best 
practices  are  not  a  silver  bullet,  it  is  impor¬ 
tant  to  understand  the  difference  between 
a  process  and  a  practice.  A  process  is  a  set 
of  interrelated  resources  and  activities  that 
transform  inputs  into  outputs.  When  used 
in  a  consistently  formal  manner,  these 
resources  and  activities  tend  to  increase 
quality,  shorten  schedules,  and  lower  cost 
and  risk.  Processes  are  used  to  conduct 
business,  and  they  support  a  unique  orga¬ 
nizational  culture.  Practices  are  disciplines, 
methods,  tools,  or  techniques  that  are  used 
to  accomplish  a  specific  function  or  set  of 
functions  in  a  project  environment.  A 
process  can  include  multiple  practices. 

A  process  can  be  considered  a  plan  in 
that  it  describes  what  must  be  done  to 
obtain  an  output  and  provides  the  frame¬ 
work  needed  to  accomplish  the  necessary 
tasks.  Practices  define  the  manner  in 
which  the  tasks  must  be  conducted.  Both 
are  critical  to  a  project’s  success  because 
what  is  to  be  done  must  be  planned,  and 
how  the  plan  will  be  accomplished  must 
be  defined.  Moreover,  practices  must  be 
adapted  to  the  organization  that  uses 
them,  and  the  relationships  between  the 
practices  must  be  understood  and  man¬ 
aged  if  the  expected  benefits  are  to  be 
realized.  It  is  incumbent  upon  the  manag¬ 
er  to  choose  practices  that  are  appropriate 
to  the  level  of  the  organization  that  will 
implement  them  and  adapt  them  as  neces¬ 
sary.  For  example,  a  practice  intended  to 
meet  the  configuration  management 
needs  of  the  acquisition  layer  of  an  organ¬ 
ization  would  not  necessarily  meet  the 
needs  of  the  development  layer,  but  with 
a  proper  understanding  of  its  uses,  a  man¬ 
ager  could  adapt  the  practice  to  meet  the 
needs  of  both. 

A  typical  program  has  several  layers. 


each  of  which  has  different  requirements 
and  constraints  regarding  the  processes 
and  practices  used  and  their  implementa¬ 
tion.  The  top  layer,  the  user  organization, 
requires  deployment  of  a  product  or  serv¬ 
ice  that  is  responsive  to  the  operational 
and  support  requirements  of  the  user 
community.  The  needs  of  the  user  com¬ 
munity  form  the  baseline  from  which 
required  practices  are  defined  and  imple¬ 
mented.  The  user  organization  requires 
practices  that  (1)  capture,  characterize,  and 
control  the  user’s  operational  require¬ 
ments  and  constraints;  (2)  define  interop¬ 
erability  and  system  interface  require¬ 
ments;  (3)  identify  how  these  require¬ 
ments  are  qualified  and  inserted  into  the 
operational  environment;  and  (4)  define 
how  these  elements  are  documented, 
maintained,  and  updated. 

The  next  layer  of  the  program,  the 
acquisition  organization,  is  chartered  to 
acquire  the  right  product  at  the  lowest  cost 
and  in  the  shortest  time  to  satisfy  a  speci¬ 
fied  user  requirement.  To  do  so,  the  acqui¬ 
sition  organization  works  with  the  user  to 
define  what  is  required.  It  then  converts 
the  user  needs  into  functional,  engineer¬ 
ing,  product  assurance  and  support 
requirements  and  constraints,  and  plans 
and  implements  processes  —  supported  by 
practices  -  to  acquire  a  system,  software 
product,  or  services  that  will  satisfy  the 
user  needs.  The  acquisition  organization 
next  prepares  and  issues  documentation 
that  establishes  agreements  between  the 
acquirer  and  supplier(s),  selects  a  suppU- 
er(s),  and  manages  the  acquisition  process 
until  the  product  or  service  is  accepted. 

The  acquisition  organization  requires 
practices  that  facilitate  the  acquisition  of 
the  product  or  service  at  the  lowest  risk.  It 
needs  practices  to  collect  and  evaluate 
quantitative  indicators  of  both  project 
performance  and  product  quality;  to 
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assure  the  product’s  quality  based  on  the 
quantitative  indicators;  to  monitor  devel¬ 
oper  performance;  and  to  facilitate  deliv¬ 
ery,  acceptance,  and  deployment  of  the 
product  or  service. 

The  next  layer  of  the  program,  the 
development  organization,  is  chartered  to 
build  a  product  in  a  manner  that  is  consis¬ 
tent  with  established  agreements  and 
specifications  and  that  maximizes  profit 
and  meets  all  commitments  and  agree¬ 
ments.  The  development  organization 
needs  practices  that  (1)  specify  architec¬ 
tures;  (2)  define,  expand,  and  control  spe¬ 
cific  engineering  requirements  that  are 
traceable  to  those  provided  by  the  acquir¬ 
er;  (3)  control  and  manage  the  develop¬ 
ment  process;  (4)  monitor  the  quality  of 
the  products  being  developed  or  received 
from  suppliers;  (5)  collect  quantitative 
information  from  implemented  practices 
to  ascertain  process  effectiveness;  (6) 
monitor  cost  and  schedule  performance; 
and  (7)  monitor  development  risk  against 
progress  toward  established  requirements. 

Finally,  support  organizations  —  such 
as  independent  test,  logistics,  installation, 
product  maintenance,  etc.  —  require  prac¬ 
tices  that  enable  processes,  and  provide 
management  visibility  into  and  control 
over  the  quality  of  the  services  provided 
and  the  risks  that  must  be  addressed. 

Given  these  various  needs,  it  is  easy  to 
see  that  a  single  set  of  best  practices 
would  be  impossible  to  implement,  and 
that  any  set  of  best  practices  must  be 
adapted  to  the  organizational  layer  that 
implements  them.  For  example,  best  prac¬ 
tices  related  to  configuration  management 
necessarily  must  be  adapted  to  meet  the 
needs  of  the  user  level,  the  supplier  level, 
and  the  development  level.  Nevertheless, 
configuration  management  can  be  consid¬ 
ered  a  necessary  best  practice  that  all 
organizations  should  address  if  they 
expect  to  succeed. 

Once  adapted  to  the  needs  of  the  var¬ 
ious  organizations,  the  practices  must  be 
integrated  if  the  project  is  to  progress 
effectively  both  within  an  organization 
and  between  organizations.  For  example, 
the  requirements  management  process 
area  may  require  effective  implementation 
of  the  requirements  definition,  configura¬ 
tion  management,  defect  identification 
and  removal,  and  user  involvement  prac¬ 
tice  areas  to  accomplish  the  process. 

In  addition,  the  many  practices  a  par¬ 
ticular  organization  uses  must  interact 
with  other  practices,  which  are  often  unre¬ 
lated,  to  provide  seamless  and  effective 
support  to  the  overall  process.  For  exam¬ 
ple,  if  a  particular  practice  identifies 
defects  early  in  the  development  process. 


configuration  management  should  ac¬ 
count  for  this,  and  the  software  manager 
must  adapt  practices  to  remove  defects 
accordingly  to  ensure  potential  efficiencies 
are  realized. 

Sample  Practices 

Table  1  on  page  16  lists  several  project 
approaches  or  best  practices  that  have 
originated  from  initiatives  conducted  by 
various  organizations.  This  list  is  a  small 
sample  of  the  hundreds  identified  in  the 
literature  as  being  best  [2].  These  practices 
can  be  categorized  by  their  typical  applica¬ 
tion  or  use: 

•  Policy  Requirement.  A  policy  re¬ 
quirement  defines  a  basic  requirement 
that  all  program  organizations  must 
meet.  This  category  focuses  on  a 
desired  outcome  and  typically  does  not 
define  specific  processes  or  practices. 

•  Organizational  Concept.  These  are 
general  principles  that  are  used  to 
organize  the  project,  allocate  resources 
and  responsibility,  enable  communica¬ 
tions,  and  effect  work  assignment. 

•  General  Strategy.  This  is  a  strategy 
that  applies  to  all  organizational  com¬ 
ponents  of  a  project  but  must  be 
adapted  to  the  needs  of  a  particular 
organization  to  be  effective.  A  general 
strategy,  for  example,  might  be  that  all 
projects  apply  continuous  risk  man¬ 
agement  to  prevent  negative  conse¬ 
quences  from  unanticipated  issues. 
Although  related  to  the  risk  manage¬ 
ment  practices  of  other  organizations, 
specific  implementation  of  the  general 
strategy  within  an  organization  will 
depend  on  the  organization’s  particular 
charter,  culture,  commitments,  and 
constraints. 

•  Business  Strategy.  This  is  a  strategy 
that  defines  how  to  accomplish  specif¬ 
ic  business  tasks. 

•  Acquisition  Framework.  This  is  a 
structure  the  acquisition  organization 
uses  to  acquire,  manage,  and  control 
the  products  or  services  and  ensure 
they  are  responsive  to  user  needs.  It 
usually  consists  of  activities,  specifica¬ 
tions,  reviews,  and  reporting  require¬ 
ments. 

•  Acquisition  Strategy.  An  overall 
statement  of  how  an  organization  will 
acquire  products  and  services  consis¬ 
tent  with  user  needs  and  requirements 
is  an  acquisition  strategy.  Expressed  in 
general  terms,  it  typically  describes 
requirements  that  will  constrain  the 
selection  and  adaptation  of  processes 
and  practices  and  defines  the  goals  that 
must  be  met  to  satisfy  validated  needs 
as  well  as  to  maximize  affordability. 


•  General  Practice.  This  is  a  practice 
that  supports  every  organizational  level 
Specific  application  of  the  practice  will 
differ  according  to  the  needs  of  the 
user,  acquirer,  supplier,  and  support  lay¬ 
ers  of  the  program  organization. 

•  Development  Practice.  This  is  a 
practice  that  predominately  supports 
the  supplier’s  requirements. 

•  Acquisition  Practice.  This  is  a  prac¬ 
tice  that  ensures  an  acquisition  is  con¬ 
ducted  effectively  by  the  acquisition 
organization  as  it  monitors  the  project 
and  controls  the  suppliers.  The  prac¬ 
tices  are  structured  to  evaluate  and 
receive  products  and  services  rather 
than  develop  and  deliver  them. 

•  Maturity  Model.  This  is  used  to  eval¬ 
uate  the  process  maturity  of  an  organ¬ 
ization  to  determine  the  potential  risk 
of  a  process  and  the  potential  to  use  it 
successfully  in  other  circumstances. 
Table  1  identifies  project  approaches, 

i.e.,  best  practices  that  have  been  extracted 
from  several  sources,  including  the 
Software  Program  Managers  Network  16 
Point  Plan,  the  DoD  Best  Practices  Study 
conducted  by  Dr.  Richard  Turner,  the 
European  Software  Institute  1977 
Software  Best  Practice  Questionnaire 
Analysis  of  Results,  and  various  other 
studies.  Some  of  the  approaches  are  relat¬ 
ed  to  policy,  others  are  related  to  process, 
and  still  others  are  related  to  strategy. 
Flowever,  they  all  are  important  consider¬ 
ations  with  significant  benefits  that  an 
organization  could  use  to  establish  an 
effective  project  environment  and  con¬ 
duct  the  activities  identified  in  their  proj¬ 
ect  plan. 

As  Norm  Brown  posits  in  IEEE 
Software  [3],  the  definition  of  a  small  num¬ 
ber  of  relevant  best  practices  can  have  a 
significant  effect  on  the  success  of  a  proj¬ 
ect,  but  only  if  the  practices  are  tailored  to 
the  needs  and  culture  of  the  organization 
that  win  use  them.  In  Table  1,  we  have 
identified  the  practice;  the  source,  includ¬ 
ing  the  primary  reference  that  was  used  to 
identify  it;  and  the  general  classification  of 
the  practice.  Turner’s  dissertation  [4], 
“Implementation  of  Best  Practices  in  U.S. 
Department  of  Defense  Software- 
Intensive  Systems  Acquisitions,”  is  identi¬ 
fied  as  the  source  for  many  of  the  prac¬ 
tices  included  in  the  table.  For  readers’ 
convenience,  we  have  included  the  source 
reference  for  the  specific  practices  identi¬ 
fied  in  the  Turner  dissertation. 

Basic  Considerations  for 
Implementing  Practices 

The  Turner  study  discusses  what  must  be 
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Best  Practices 


Practice  Identification 

Practice  Source 

Practice  Category 

Establish  Clear  Goals  and  Decision  Points 

(DSBTF,  5000.2R)  [5,6] 

Policy  Requirement 

Treat  People  as  the  Most  Important  Resource 

SPMN  16  Point  Plan  [7] 

Policy  Requirement 

Common  Management  and  Manufacturing  Systems 

(5000. 2 R)  [6] 

Policy  Requirement 

Integrated  Product  and  Process  Development 

(5000.2R,  Reifer)  [6,4] 

Organizational  Concept 

Appointing  Project  Managers  for  Each  Project 

ESI  [8] 

Organizational  Concept 

Software  Quality  Assurance  Function  with  Independent  Reporting  Line 

ESI  [8] 

Organizational  Concept 

Training  New  Project  Managers 

ESI  [8] 

Organizational  Concept 

Have  a  Formal  Review  or  Handover  of  Deliverables  From  One  Project  Group  to  Another 

ESI  [8] 

Organizational  Concept 

Maintaining  Awareness  of  New  Development  Technologies 

ESI  [8] 

General  Strategy 

Ensuring  User  Input  at  All  Stages  of  the  Project 

ESI  [8] 

General  Strategy 

Assess  Viability,  Risks,  and  Benefits  Before  Committing  to  a  Project 

ESI  [8] 

General  Strategy 

Conduct  Periodic  Reviews  of  the  Status  of  Projects 

ESI  [8] 

General  Strategy 

Demonstration-Based  Reviews  (Including  Executable  Architectures) 

(Royce,  DSBTF,  ISO)  [4,5,9] 

General  Strategy 

Conduct  Inspection  and  Walkthroughs  at  Each  Stage 

ESI  [8] 

General  Strategy 

Require  Structured  Development  Methods  (Iterative  Processes) 

(Royce,  DSBTF)  [4,5] 

General  Strategy 

Plan  for  Technology  Insertion 

(5000.2R,  DSMC)  [6,10] 

General  Strategy 

Commercial  and  Specifications  and  Standards/Open  Systems 

(5000. 2R,  Anderson,  Jones)  [6,4] 

General  Strategy 

Statistical  Process  Control 

Reifer  [4] 

General  Strategy 

Capture  Artifacts  in  Rigorous,  Model-Based  Notation 

(Royce,  DSBTF)  [4,5] 

General  Strategy 

Strategic  Partnering 

Reifer  [4] 

Business  Strategy 

Relationship  Management 

Reifer  [4] 

Business  Strategy 

Market  Watch 

Reifer  [4] 

Business  Strategy 

Enterprise-Wide  Licensing 

Reifer  [4] 

Business  Strategy 

Independent  Expert  Reviews/SCEs 

(5000.2R,  DSBTF,  DSMC,  Jones) 
[6,5,10,4] 

Acquisition  Framework 

Performance-Based  Specifications 

(5000.2R,  Anderson,  Reifer)  [6,  4] 

Acquisition  Framework 

Use  Past  Performance 

(5000.2R,  DSBTF,  Anderson,  Reifer) 
[6,5,4] 

Acquisition  Framework 

Leverage  Commercial  Off-the-Shelf  (COTS  Items,  Non-Developmental  Items  (NDI) 

(Anderson,  Reifer)  [4] 

Acquisition  Framework 

Ensure  that  Subcontractors  Follow  Formal  Processes 

ESI  [8] 

Acquisition  Strategy 

Best  Value  Awards 

(5000.2R,  DSBTF,  Anderson,  Reifer) 
[6,5,4] 

Acquisition  Strategy 

Adopt  Continuous  Risk  Management 

SPMN  16  Point  Plan  [7] 

General  Practice 

Estimate  Empirically  Cost  and  Schedule 

SPMN  16  Point  Plan  [7] 

General  Practice 

Use  Metrics  to  Manage 

SPMN  16  Point  Plan  [7] 

General  Practice 

Track  Earned  Value 

SPMN  16  Point  Plan  [7] 

General  Practice 

Track  Defects  Against  Quality  Targets 

SPMN  16  Point  Plan  [7] 

General  Practice 

Adopt  Life  Cycle  Configuration  Management 

SPMN  16  Point  Plan  [7] 

General  Practice 

Manage  and  Trace  Requirements 

SPMN  16  Point  Plan  [7] 

General  Practice 

Ensure  Data  and  Database  Operability 

SPMN  16  Point  Plan  [7] 

General  Practice 

Assess  Reuse  Risks  and  Costs 

SPMN  16  Point  Plan  [7] 

General  Practice 

Inspect  Requirements  and  Design 

SPMN  16  Point  Plan  [7] 

General  Practice 

Requirements  Trade-Off/Negotiation 

(DSBTF,  Royce)  [5,4] 

General  Practice 

Perform  Independent  Testing 

ESI  [8] 

General  Practice 

Have  Formal  Methods  of  Estimating  Software  Size 

ESI  [8] 

General  Practice 

Formally  Review  the  Functionality  of  the  System  the  Software  Replaces 

ESI  [8] 

General  Practice 

Use  Formal  Methods  to  Estimate  Schedule  and  Cost 

ESI  [8] 

General  Practice 

Use  System-Based  Software  Design 

SPMN  16  Point  Plan  [7] 

Development  Practice 

Define  and  Control  Interfaces 

SPMN  16  Point  Plan  [7] 

Development  Practice 

Design  Twice,  Code  Once 

SPMN  16  Point  Plan  [7] 

Development  Practice 

Manage  Testing  as  a  Continuous  Process 

SPMN  16  Point  Plan  [7] 

Development  Practice 

Compile  and  Smoke  Test  Frequently 

SPMN  16  Point  Plan  [7] 

Development  Practice 

Architecture-First  Approach 

(Royce,  DSMC,  DSBF)  [4,10,  5] 

Development  Practice 

Have  Common  Coding  Standards  for  Projects 

ESI  [8] 

Development  Practice 

Plan  Testing  Before  Coding 

ESI  [8] 

Development  Practice 

Establish  Reliability  and  Stress  Margins 

Doherty  SEI  [11] 

Acquisition  Practice 

Personal  Software  ProcessS“/Team  Software  Process^M  Practices 

Humphrey  [12,13] 

Maturity  Model 

Acquisition  Process  Improvement 

SEI  [14] 

Maturity  Model 

Contractor  Capability  Evaluation 

SEI  [15] 

Maturity  Model 

DSBTF:  Defense  Science  Board  Task  Force 
ESI;  Enterprise  Software  Initiative 
SCE:  Software  Capability  Evaluation 
DSMC:  Defense  Systems  Management  College 


Team  Software  Process  and  Personal  Software  Process  are  service  marks  of  Carnegie  Mellon  University. 


Table  1 :  Examples  of  Best  Practices 
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considered  to  successfully  migrate  prac¬ 
tices  from  project  to  project.  Before  man¬ 
agement  selects  specific  practices  to 
accomplish  a  project,  it  must  address  the 
organization’s  culture,  attitude,  and  experi¬ 
ence,  and  most  certainly  these  factors 
must  be  considered  when  management 
devises  its  strategy  to  introduce  the  prac¬ 
tices  selected  [16], 

For  example,  early  identification  and 
removal  of  defects  through  consistent 
application  of  structured  inspections  is  a 
valuable  goal.  However,  if  the  organiza¬ 
tion’s  management  presumes  that  its  engi¬ 
neers  win  not  make  mistakes,  and  if  it  will 
not  adequately  fund  this  practice  consis¬ 
tently  from  concept  through  delivery,  the 
practice  is  not  realistic  given  the  organiza¬ 
tion’s  culture. 

Not  only  must  management  under¬ 
stand  its  organization’s  culture,  it  must 
also  understand  the  true  costs  and  risks 
associated  with  implementing  a  practice 
before  it  commits  to  using  the  practice. 
That  is,  management  must  honestly  assess 
and  understand  the  following: 

•  The  effect  of  the  practice  on  project 
teams  regarding  their  possible  resist¬ 
ance  and  the  potential  for  increased 
productivity. 

•  The  costs  associated  with  the  practice 
and  the  potential  return  on  invest¬ 
ment. 

•  The  cost  required  to  train  those  who 
will  apply  the  practice. 

•  The  availability  and  cost  of  associated 
tools. 

•  Potential  barriers  to  implement  the 
practice  and  its  application. 

•  The  validity  and  general  acceptance  of 
the  practice  within  the  industry. 

•  The  effect  of  the  practice  on  related 
and  interfacing  practices,  processes, 
and  tools. 

•  The  degree  of  management  and  staff 
commitment  to  the  practice,  and  what 
factors  led  them  to  commit  to  the 
practice. 

It  is  critical  that  management  under¬ 
stand  the  true  costs  and  impacts  of  the 
practices  it  implements,  whether  they  have 
been  proven  in  other  environments  or 
not.  If  management  implements  a  practice 
without  understanding  its  costs  and 
effects,  it  could  well  be  incompletely  or 
haphazardly  implemented,  and  the  project 
wiU  suffer  as  a  result.  Indeed,  if  the  imple¬ 
mentation  is  incomplete,  poorly  planned, 
or  otherwise  improper,  or,  worse,  if  the 
practice  must  be  replaced  mid-project,  the 
effects  -  such  as  poor  staff  morale  and 
productivity,  tool  replacement,  retraining, 
and  file  or  artifact  conversion  -  can  be 
devastating. 


In  addition  to  these  considerations,  proj¬ 
ects  with  substantial  software  content  that 
show  evidence  of  certain  characteristics  are 
poor  candidates  for  the  reasoned  applica¬ 
tion  of  best  practices.  These  characteristics, 
which  were  documented  in  the  April  2002 
Crosstalk  [IT],  are  the  following: 

•  Unwarranted  optimism  and  unrealistic 
executive  management  expectations. 

•  Late  decision-making. 

•  Inappropriate  use  of  the  standard 
software  process. 

•  Missing  or  inadequately  implemented 
program  activities. 

•  Lack  of  leadership. 

•  Early  declarations  of  victory. 

•  An  absence  of  risk  management, 
which  could  convince  managers  and 
staff  they  can  accomplish  unrealistic 
objectives  given  the  actual  project  cir¬ 
cumstances. 

These  underlying  attitudes,  which  can 
be  understood  to  presume  project  success 
and  minimal  risk,  often  convince  project 
management  and  staff  that  they  need  not 
adequately  plan  to  implement  a  practice 
and  develop  process  standards.  Such  com¬ 
placency  can  be  costly. 

Given  the  fact  that  a  best  practices  sil¬ 
ver  bullet  does  not  exist,  organizations 
cannot  unthinkingly  adopt  a  pro  forma 
approach  to  project  completion  and 
assume  the  practices  they  implement  will 
automatically  succeed.  To  truly  succeed, 
management  must  understand  how  the 
practices  they  use  wiU  work  within  their 
unique  organization,  which  will  lead  to  a 
soUd  project  management  foundation  and 
will  in  turn  positively  affect  the  bottom 
Une  regarding  productivity,  quaUty,  timeU- 
ness,  and  user  satisfaction.^ 
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Measure  Like  a  Fighter  Pilot 

Joe  H.  Lindley 
Buijtheon 


The  late  Col.  John  Bojd,  U.S.  Air  Force,  developed  the  Observe,  Orient,  Decide,  and  Act  (OODA)  Foop  as  a  strategy  for 
air  combat  and  warfare.  It  is  an  elegant,  fast  decision-loop  strategy  that  is  also  useful  in  engaging  the  measurement  and  analy¬ 
sis  process  with  the  corrective  action  loops  of  a  development  project.  The  OODA  Toop  will  be  introduced  in  this  article  along 
with  practical  suggestions  and  lessons  learned  related  to  implementation  of  the  measurement  and  analysis  process.  These  rec¬ 
ommendations  are  based  upon  years  of  measurement  and  analysis  experience,  recently  with  a  Capability  Maturity  ModeF 
Integration  Tevel  3  Best  Practice  measurement  and  analysis  process. 


A  measurement  analyst  can  face  daunt¬ 
ing  challenges  when  implementing  a 
measurement  and  analysis  process.  Project 
dynamics  and  the  array  of  forces  that 
oppose  change  can  easily  stall  implementa¬ 
tion,  even  when  the  process  is  based  upon 
respected  guidelines  such  as  Capability 
Maturity  Model®  Integration  (CMMI®).  To 
be  successful,  a  measurement  analyst  must 
have  an  effective  game  plan  (a  focused 
strategy  of  when  to  do  what,  where). 

After  years  of  experience  with  meas¬ 
urement  and  analysis,  most  recently  devel¬ 
oping  a  CMMI  Level  3  Best  Practice  meas¬ 
urement  program  for  a  large  development 
and  integration  program,  I  have  accumulat¬ 
ed  a  number  of  suggestions  and  lessons 
learned.  I  did  not  have  a  cogent  game  plan 
that  I  could  share  with  others,  however, 
until  I  discovered  the  Observe,  Orient, 
Decide,  and  Act  (OODA)  Loop,  which 
explained  almost  aU  the  successes  and  fail¬ 
ures  I  have  experienced. 

The  steps  of  this  decision-loop  con¬ 
cept  provide  a  forward-leaning  framework 
for  action  that  wiU  bring  measurement  and 
analysis  to  its  full  potential.  The  OODA 
Loop  wiU  be  used  in  this  article  as  a  frame¬ 
work  to  relate  my  first-hand  suggestions 
and  lessons  learned. 

The  OODA  Loop 

Col.  John  R.  Boyd,  an  accomplished  U.S. 
Air  Force  fighter  pilot  and  military  thinker, 
originally  developed  the  OODA  Loop  as  a 
winning  strategy  for  air  combat.  Later,  it 
was  a  key  part  of  Boyd’s  maneuver  warfare 
strategy,  which  has  been  embraced  at  some 
level  by  most  U.S.  armed  services.  The 
phrase  getting  inside  the  enemy ’s  decision  loop  can 
be  heard  in  military  briefings  in  recent 
years  -  a  reference  to  the  OODA  Loop. 
Col.  Boyd  passed  away  in  1997  at  the  age 
of  70. 

OODA  Loop  Definition 

The  OODA  Loop  is  a  strategy  for  effective 
decision  making.  Figure  1  is  a  simplified 


view  of  the  concept.  Boyd  broke  the  deci¬ 
sion-making  process  into  four  steps: 
Observe,  Orient,  Decide,  and  Act,  as 
defined  below: 

1.  Observe.  Collect  and  organize  infor¬ 
mation  related  to  recent  events,  feed¬ 
back  from  prior  actions,  and  changes  in 
the  environment. 

2.  Orient.  Orient  to  the  unfolding  envi¬ 
ronment  and  current  observations. 
This  is  the  most  critical  step  and  is 
prone  to  failure.  The  speed  with  which 
orientation  is  completed  is  dependent 
upon  the  current  paradigm  (i.e.,  expec¬ 
tation  of  what  the  observations  and 
current  environment  should  be).  In 
simplistic  terms,  the  Orient  step  wiU 
complete  in  one  of  three  ways.  Each  is 
characterized  by  the  speed  with  which 
orientation  wiU  complete: 

•  Fast.  Observations  and  environ¬ 
ment  match  expectations  (para- 
digm):  Quick,  gut-feel,  decision/ 
action  can  be  initiated. 

•  Moderate.  Observations  and  envi¬ 
ronment  differ  in  some  ways  from 
the  paradigm:  Analysis  wiU  be 
required  to  develop  alternate  plans 
of  action  and  the  paradigm  may 
need  to  be  adjusted. 

•  Slow.  Observations  and  environ¬ 
ment  differ  substantiaUy  from  the 
paradigm:  The  current  paradigm 
may  have  to  be  discarded  and  a  new 

Figure  1:  The  OODA  Loop 


one  synthesized. 

3.  Decide.  From  available  actions,  select 
which  one  to  take. 

4.  Act.  Take  action. 

Boyd  also  asserted  the  foUowing: 

•  AU  human  behavior,  individual  or  orga¬ 
nizational,  can  be  depicted  as  a  contin¬ 
ual  cycling  through  the  four  steps. 

•  Success  or  failure  depends  upon  the 
relationship  of  one’s  own  loop  to  that 
of  an  opposing  loop. 

•  Success  or  failure  depends  on  one’s 
abiUty  to  swifdy  and  accurately  com¬ 
plete  cycles  of  one’s  own  loop  faster 
than  opposing  forces  can  cycle  through 
their  loop. 

Jet  Fighter  Example 

The  utiUty  of  the  OODA  Loop  is  best 
iUustrated  by  an  example  from  Boyd’s  com¬ 
bat  experience  in  the  Korean  War  -  a  dog¬ 
fight  between  a  U.S.  F-86E  Sabre  Jet  and  a 
Russian  MiG-15.  Boyd  was  interested  in  a 
paradox  related  to  these  fighters.  Sabre  Jets 
were  shooting  down  MiGs  at  a  10-1  ratio 
in  the  Korean  War,  in  spite  of  the  fact  that 
MiGs  could  out-run  and  out-turn  Sabre 
Jets.  The  Sabre  Jet  provided  better  visibiU- 
ty  and  was  more  responsive  than  the  MiG, 
but  this  was  not  enough  to  explain  its  suc¬ 
cess  ratio  until  the  OODA  Loop  concept 
demonstrated  the  power  of  a  fast  decision 
loop. 

Each  Sabre  Jet  maneuver  is  creating  a 
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Figure  2:  Shallow  Measurement  Ijoops  Vail 


change  in  the  tactical  situation.  If  the  MiG 
pilot  is  completing  maneuvers  (and 
OODA  Loop  cycles)  at  the  same  pace,  he 
will  be  able  to  observe  and  reorient  to  each 
maneuver  as  it  occurs.  New  observations 
win  generally  match  his  expectations  since 
he  is  reorienting  to  each  new  maneuver. 
His  performance  should  therefore  be  on 
par  with  the  Sabre  Jet  pilot. 

A  trained  Sabre  Jet  pilot,  however,  will 
use  his  jet’s  responsiveness  to  complete 
maneuvers  faster  than  the  MiG.  The  MiG 
pilot,  now  unable  to  cycle  through  the  loop 
as  fast  as  the  Sabre  Jet,  will  fail  to  keep  his 
orientation  updated,  and  wiU  begin  to 
notice  a  mismatch  between  what  he 
expects  to  observe  and  what  he  actually 
observes.  Resolving  the  ambiguity  of  the 
mismatch  wiU  slow  down  the  MiG  pilot’s 
orientation  forcing  him  to  cycle  through 
the  loop  even  slower,  leading  to  larger  mis¬ 
matches.  For  the  MiG  pilot,  time  wiU 
appear  to  be  compressed,  giving  him  Uttle 
time  to  think.  For  the  Sabre  Jet  pilot,  time 
wiU  seem  to  almost  stretch  out  and  slow 
down.  The  Sabre  pilot  wUl  be  able  to 
harass  and  eventuaUy  entrap  the  MiG  pilot 
to  win  the  engagement. 

The  Sabre  Jet  pilot  can  win  by  using 
fast  transient  maneuvers  to  degrade  the 
MiG  pilot’s  orientation.  For  air  combat  and 
warfare,  speeding  up  one’s  own  OODA 


Loop  (operating  within  the  enemy’s 
OODA  Loop)  can  be  used  to  UteraUy 
destroy  an  enemy’s  decision-making  capa- 
bUity 

On  the  other  hand,  the  same  idea  can 
be  used  constructively  for  engineering 
process.  By  operating  within  the  OODA 
Loop  of  a  target  process,  a  process  change 
leader  can  make  supportive  actions  to 
enhance  and  positively  influence  the  target 
process  decision  loop. 

The  OODA  Loop  as  an 
Enabler  for  Measurement 
and  Analysis 

Current  development  processes  such  as 
CMMI  and  Practical  Software  and  Systems 
Measurement  provide  exceUent  guidance 
on  how  to  plan  and  conduct  measurement 
and  analysis.  A  critical  chaUenge  in  imple¬ 
menting  these  models,  however,  is  interact¬ 
ing  with  other  project  processes;  in  other 
words,  engaging  with  the  managers  and 
other  stakeholders  who  actually  initiate 
corrective  action  -  the  end  goal  for  meas¬ 
urement  and  analysis. 

We  have  aU  seen  projects  that  have  metrics 
but  practice  a  shallow  implementation, 
where  metrics  are  coUected  and  reported 
but  do  not  have  much  of  an  impact  on  the 
stakeholders  or  drive  corrective  action  at  aU. 


Figure  3:  Deep  Measurement  Voops  Engage  and  Succeed 


Figure  2  Ulustrates  this  situation.  Stake¬ 
holders  in  this  environment  generally  fail  to 
appreciate  the  value  of  measurement. 

On  the  contrary,  the  measurement  loop 
should  extend  deep  into  the  target  process. 
Figure  3  illustrates  the  ideal  situation.  With 
a  deep  measurement  loop,  the  target 
processes  wiU  depend  upon  measures,  and 
the  measures  wiU  regularly  drive  or  shape 
change.  In  this  situation,  the  measures  truly 
engage  target  processes.  Target  process 
stakeholders  wiU  request  more  measure¬ 
ments  over  time,  raising  the  importance  of 
measurement  in  their  project  activities.  At 
this  level  of  engagement  with  its  target 
process,  measurement  and  analysis  is  excit¬ 
ing,  vital  work. 

Driving  a  measurement  loop  deep  into 
the  target  process  requires  a  focused  strat¬ 
egy.  My  suggestion  is  to  engage  the  target 
process  by  getting  inside  its  decision  loop. 
Visualize  the  decision  loop  that  stakehold¬ 
ers  use  for  their  decisions  (Observe, 
Orient,  Decide,  and  Act)  and  include 
measurement  collection  and  analysis  in  the 
Observe  and  Orient  steps  (see  Figure  4). 
Consider  this  the  playing  field  for  the 
measurement  analyst.  All  the  measures 
provided  should  be  weighed  against  how 
they  wUl  play  in  the  loop.  The  measure¬ 
ment  analyst  should  drive  his/her  own 
decision  loop  fast  enough  to  stay  inside  the 
stakeholders’  decision  loop  to  deliver  what 
they  need  when  they  need  it.  The  measure¬ 
ment  analyst  should  be  making  fast  tran¬ 
sient  moves,  like  a  Sabre  Jet  pilot,  to 
enhance  the  stakeholder  decision  loop. 

Getting  inside  the  stakeholders’  deci¬ 
sion  loop  may  be  a  significant  paradigm 
shift  for  some  measurement  analysts.  It  wiU 
puU  them  away  from  the  relative  safety  of 
the  numbers  and  into  the  project  where 
they  wiU  have  to  find  out  how  to  reaUy 
drive  and  shape  change.  Measurement  wiU 
become  more  chaUenging,  but  much  more 
effective. 

Practical  Suggestion 

A  sign  that  you  are  truly  inside  your  stake¬ 
holders’  decision  loop  is  the  interaction 
with  the  stakeholders.  They  wiU  ask  ques¬ 
tions,  request  and  even  argue  over  refine¬ 
ments,  and  take  a  real  interest  in  your 
measures.  During  my  measurement  brief¬ 
ings  to  our  customer,  my  project  managers 
often  add  commentary  of  their  own.  They 
have  played  a  significant  role  in  refining  the 
measures  and  are  very  famiUar  with  my 
charts. 

The  OODA  Loop  Applied  to 
Measurement  and  Analysis 

The  OODA  Loop  is  used  in  measurement 
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and  analysis  as  a  framework  for  action. 
Illustrated  in  Figure  5,  the  OODA  Loop 
cycle  includes  the  measurement  and  analy¬ 
sis  functions  in  the  observe  phase  with  the 
remainder  of  the  loop’s  actions  (Orient, 
Decide,  and  Act)  primarily  in  the  target 
process.  The  Orient  step  is  a  joint  effort, 
wherein  the  measurement  analyst  orients 
the  observations  to  the  stakeholders’  per¬ 
spectives,  and  the  stakeholders  then  famil¬ 
iarize  themselves  to  the  unfolding  project 
environment. 

Note  that  the  figure  represents  a  meas¬ 
urement  and  analysis  loop  for  a  single  proj¬ 
ect  activity.  For  a  project,  many  of  these 
win  exist.  The  diagram  includes  three 
corollaries  associated  with  the  OODA 
Loop  concept,  which  will  be  defined  in  the 
Corollaries  section:  tempo,  harmony,  and 
ground  truth. 

Each  of  the  OODA  Loop  steps 
(Observe,  Orient,  Decide,  and  Act)  is  de¬ 
fined  below,  along  with  related  practical 
suggestions  and  lessons  learned. 

Observe 

The  observe  step  for  the  measurement 
analyst  entails  the  collection  and  analysis  of 
measurement  data. 

Practical  Suggestion 

One  of  the  most  difficult  and  valuable 
services  a  measurement  analyst  can  pro¬ 
vide  is  forecasting  (e.g.,  forecasting  the 
expected  number  of  defects  that  will  be 
encountered  during  formal  test).  It  is  diffi¬ 
cult  because  forecasting  requires  creative 
analysis  and  data  collection  from  a  variety 
of  sources.  Forecasting  is  especially  valu¬ 
able  because  it  provides  a  basis  for  objec¬ 
tively  assessing  status,  finding  leading  indi¬ 
cators,  and  confirming  management 
assumptions. 

Orient 

Process  stakeholders  perform  the  Orient 
step  by  orienting  to  the  ongoing  situation 
in  preparation  for  the  last  steps  of  the 
loop.  Decide  and  Act.  The  measurement 
analyst  plays  a  key  role  in  the  step  by  ori¬ 
enting  the  measures  for  the  stakeholders 
(i.e.,  making  the  data  meaningful  in  a  per¬ 
spective  that  is  both  familiar  and  effective 
in  decision  making) . 

Figure  5  includes  an  extra  paradigm 
loop  for  the  Orient  step  related  to  the 
stakeholder  paradigm.  This  is  an  important 
concept.  Based  on  experience  and  attitude, 
each  stakeholder  will  have  an  expectation 
of  how  measurement  should  work,  and 
what  the  measurements  will  be.  This  is  the 
stakeholder  paradigm.  It  will  have  an 
impact  on  how  the  stakeholder  reacts  to 
measures  and  can  either  help  or  hinder 


measurement.  The  measurement  analyst 
win  generally  need  to  structure  measures 
so  that  they  are  aligned  with  the  stakehold¬ 
er  paradigm  and,  occasionally,  work  on 
modifying  the  stakeholder  paradigm. 

Practical  Suggestion 

Provide  status  data  for  the  whole  organi¬ 
zation  plus  driU-downs  (filtering  or  other 
methods  of  providing  status  for  lower- 
level  organizations).  See  Figure  6  for  an 
example.  This  format  is  used  to  brief 
code-size  status  for  the  project  (top  left 
chart  in  Figure  6)  and  each  of  the  lower- 
level  subsystems.  In  this  way,  problems  at 
the  subsystem  level  (such  as  shown  in  the 
bottom  right  chart  of  Figure  6)  are  not 
missed.  The  charts  are  simple  and  provide 
limits/plans  so  that  reviewers  can  objec¬ 
tively  assess  status. 

Lesson  Learned 

I  once  failed  to  ensure  that  project  man¬ 
agers  reviewed  measurement  data  at  the 
normal  periodic  rate.  The  managers  were 
busy  and  I  did  not  see  anything  to  worry 
about,  so  I  did  not  insist  on  the  normal 


Figure  4:  Engage  by  Getting  Inside  the  Target 
OODA  Eoop 


measures  reviews.  I  realized  later  that  we 
had  missed  an  opportunity  to  take  correc¬ 
tive  action  on  a  problem  the  managers 
would  have  spotted  if  they  had  seen  the 
data.  I  had  failed  to  recognize  that  it  was 
their  orientation,  not  mine,  that  was  impor¬ 
tant. 

Decide 

Project  stakeholders  perform  this  step.  The 
challenge  for  the  measurement  analyst  is  to 
provide  measures  that  will  support  reliable 
and  expeditious  decision  making  by  the 
stakeholders. 

Practical  Suggestion 

To  optimize  decision  making,  I  follow 
what  I  call  the  Bruno  the  Trained  Ape  rule  for 


Figure  5:  The  OODA  Eoop  for  Measurement  and  Analysis 


Figure  6:  Example  Code  Sit^e  Charts 


Project  Code  Size 


Subsystem  Code  Size 


CJ 


Subsystem  Code  Size 


Subsystem  Code  Size 


CG 


September  2005 


www.stsc.hill.af.mil  2 1 


Software  Engineering  Technology 


charts.  Charts  must  be  so  simple  that  an 
ape  can  understand  them.  This  is  an  exag¬ 
geration  of  course,  and  has  nothing  to  do 
with  the  intelligence  of  stakeholders.  The 
issue  is  that  busy  stakeholders  only  have  a 
few  seconds  to  assess  a  chart.  The  answer 
needs  to  jump  out  at  them.  Charts  should 
be  stripped  of  anything  that  is  not  directiy 
related  to  the  decision  at  hand. 

Lesson  Learned 

In  terms  of  chart  design  and  its  impor¬ 
tance  in  decision  making,  I  have  learned  a 
great  deal  from  the  books  by  Edward  Tufte 
[1,  2].  He  provides  profound  suggestions 
and  some  interesting  examples  of  lessons 
learned  related  to  the  importance  of  trim¬ 
ming  charts  down  to  only  meaningful  data. 

Act 

As  with  the  Decide  step,  this  step  is  taken 
by  project  stakeholders,  so  the  challenge 
for  the  measurement  analyst  is  to  provide 
actionable  measures  that  support  expedi¬ 
tious  corrective  action.  An  analysis  that 
leads  to  a  decision  to  act  is  often  useless 
unless  a  suitable  corrective  action  is  identi¬ 
fied  immediately  or  shortly  thereafter. 

Practical  Suggestion 

I  publish  integration  defect  spreadsheets 
every  week  to  assist  stakeholders  in  assess¬ 
ing  the  status  of  defect  resolution.  A  well- 
known  part  of  these  spreadsheets  is  what 
the  leads  call,  in  jest,  the  flogging  list.  It  is 
a  listing  of  defects  with  a  subsystem  filter, 
which  win  display  the  details  for  all  defects 
belonging  to  a  selected  subsystem.  At 
weekly  meetings,  subsystem  leads  generally 
report  status  using  the  flogging  list  so  that 
the  project  manager  has  the  data  needed  to 
allocate  resources  and  take  immediate  cor¬ 
rective  action.  The  flogging  list  is  a  good 


example  of  an  actionable  measure. 

Corollaries 

The  framework  for  action  established  by 
the  OODA  Loop  lends  itself  to  the  devel¬ 
opment  of  corollaries  —  valuable  principles 
related  to  the  OODA  Loop  that  I  have 
used  successfully.  Three  corollaries  are 
defined  below:  tempo,  harmony,  and 
ground  truth.  Figure  5  depicts  the  OODA 
Loop  and  the  context  of  these  corollaries. 

Tempo 

Tempo  is  the  goal  that  OODA  Loop  tim¬ 
ing  must  match  or  exceed  the  timing  of  the 
target  process.  The  OODA  Loop  tech¬ 
nique  win  fail  totally  unless  tempo  is  ade¬ 
quate.  This  includes  both  the  measurement 
cycle  time  (how  often  the  loop  is  executed) 
and  the  loop  cycle  time  (how  quickly  each 
loop  cycle  is  completed). 

Practical  Suggestion 

The  measurement  and  analysis  OODA 
Loop  must  operate  within  the  natural  tim¬ 
ing  of  the  process  being  measured: 

•  Be  aware  that  loop  cycle  times  may  be 
surprisingly  short  —  even  with  long 
measurement  cycles. 

•  Develop  tools  (e.g,  macros,  scripts, 
queries)  to  gain  efficiency  and  reduce 
the  time  it  takes  to  respond  to  ad-hoc 
management  requests. 

•  Take  advanced  courses  on  spread¬ 
sheets,  databases,  etc.  to  gain  proficien¬ 
cy  in  tool  development. 

Lesson  Learned 

I  learned  the  value  of  tempo  the  hard 
way.  By  the  time  I  completed  a  full  analy¬ 
sis  of  our  inspection  process  for  our  first 
increment,  we  were  halfway  through  cod¬ 
ing  on  the  next  increment  —  too  late  to 


help  with  that  increment. 

Harmony 

Harmony  is  the  goal  of  maintaining  a 
good  relationship  between  the  measure¬ 
ment  analyst  and  the  project  personnel 
that  provide  measurement  data.  Without 
harmony,  data  collection  will  be  difficult, 
and  corrective  actions  may  not  be  con¬ 
structive. 

Practical  Suggestions 

To  develop  a  good  working  relationship 
with  data  providers,  consider  the  follow¬ 
ing: 

•  Try  to  use  data  that  is  easy  for  the  data 
providers  to  obtain. 

•  Personally  visit  data  providers  whenev¬ 
er  possible  to  develop  a  good  working 
relationship. 

•  Discuss  potential  corrective  action 
with  data  providers  to  emphasize  the 
real  goal  of  measurement. 

•  Never  brief  negative  information 
without  first  informing  the  involved 
stakeholders  and  including  their  per¬ 
spective  in  the  briefing. 

Ground  Truth 

Ground  Truth  is  the  goal  of  establishing 
project  measurement  archives  and  arti¬ 
facts  that  are  regarded  by  project  person¬ 
nel  as  highly  reliable,  comprehensive,  and 
detailed.  To  attain  this  goal,  the  archives 
and  artifacts  must  also  be  readily  accessi¬ 
ble  and  user  friendly. 

Practical  Suggestion 

Data  should  reach  deep  into  the  organiza¬ 
tion,  providing  a  fingertip  feel  of  the  situ¬ 
ation.  See  Figure  7  for  an  example  of  an 
approach  to  meet  this  need.  Resolving 
integration  defects  on  a  large  project 
requires  the  coordinated  effort  of  several 
teams  of  professionals. 

Defects  flow  through  different  states 
(e.g.,  review,  fix,  or  verify)  as  they  are 
processed  (worked  off)  by  different  teams. 
If  defects  fall  to  flow  as  expected  deep 
within  the  project,  bottlenecks  can  occur 
that  can  slow  the  entire  effort.  With  this 
driU-down  type  chart,  the  user  can  specify 
a  bin  (group  of  defects  related  to  specific 
baselines,  states,  and/or  teams)  to  be  dis¬ 
played  and  view  the  defect  flow  through 
the  bin.  The  defect  flow  is  indicated  by 
three  plots  on  the  chart: 

•  Cumulative  In.  Cumulative  number 
of  defects  that  have  entered  the  bin, 
which  rises  over  time  as  new  defects 
flow  into  the  bin. 

•  Cumulative  Out.  Cumulative  num¬ 
ber  of  defects  worked  off,  which  rises 
over  time  as  defects  in  the  bin  are 


Figure  7:  Hxampk  Defect  Chart 
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worked  off. 

•  Backlog.  Current  number  of  defects 
being  worked  within  the  bin,  which 
will  remain  constant  over  time  if 
defects  are  worked  off  as  fast  as  they 
flow  into  the  bin. 

The  plots  answer  status  questions  for  the 
bin  such  as,  “Is  the  team  keeping  up  with 
the  flow  of  defects?”  or  “Why  is  the  back¬ 
log  rising?” 

Conclusion 

Using  the  measurement  and  analysis 
OODA  Loop  to  engage  a  project  creates  a 
focused,  fast-paced  measurement  process, 
drives  corrective  action,  and  brings  meas¬ 
urement  and  analysis  to  its  full  potential. 
The  OODA  Loop  also  imparts  a  high- 
energy,  spirited  tone  to  measurement  by 
keeping  the  focus  on  the  project  dynamics 
of  decision  making.  The  result  is  a  suc¬ 
cessful  and  rewarding  process  exper¬ 
ience. ♦ 
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on  programs.  PSM  is  based  on  actual 
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version  of  the  PSM  Guidebook. 


Coming  Events 


October  16-20 

Object-Oriented  Programming,  Systems, 
Languages,  and  Applications  Conference 
San  Diego,  CA 

www.oopsla.org/2005/ShowPage. 

do?id=Home 

October  17-20 

MILCOM  2005 

Military  Communication  Conference 
Atlantic  City,  NJ 

www.milcom.org/2005 

October  17-21 

Quality  Assurance  Institutes  26^ 
Annual  Sofiware  Testing  Conference 
Orlando,  FL 

www.qaiusa.com 

October  24-27 

5'*  Annual 

Systems  Engineering  Conference 
San  Diego,  CA 

http://register.ndia.org/interview/ 

register.ndia? 

November  6-9 

Annual  Amplifying  Your 
Effectiveness  Conference 
Phoenix,  AZ 

www.ayeconference.com 

November  13-17 

International  Association  for  Computing 
Machinery  SIGAda  2005  Conference 
Atlanta,  GA 

www.sigada.org/conf/sigada2005 

November  14-18 

STARWEST2005 
Anaheim,  CA 

www.sqe.com/starwest 

May  1-4,2006 

2006  Systems  and  Software 
Technology  Conference 

\Jysiems  &  Software 
Technology  Conference 

Salt  Lake  City,  UT 

www.stc-online.org 


September  2005 


www.stsc.hill.af.mil  23 


Applying  Functional  TSP  to  a  Maintenance  Project 


Ellen  George  and  Dr.  Steve  Janiszewski 
PS<&J  Software  Six  Sigma 

Personal  Software  Process™  (PSP™)  is  taught  in  the  context  of  new  code  development  and  implementation.  We  frequently 
hear  developers  say  that  it  will  only  work  for  new  development  and  that  their  maintenance  project  is  different.  This  article  dis¬ 
pels  that  myth,  demonstrating  how  PSP  and  Team  Software  Process™  (TSP™)  were  successfully  adapted  to  plan  and  man¬ 
age  a  maintenance  project.  Teaders  will  learn  the  key  success  factors  in  forming  a  functional  project  team  and  will  learn  how 
to  emphasise  and  de-emphasige  portions  of  the  TSP  process  to  better  address  the  needs  of  a  maintenance  project. 


The  Personal  Software  Process™  (PSP'‘^“) 
is  a  software  development  process  orig¬ 
inated  by  Watts  Humphrey  at  the  Software 
Engineering  Institute  (SEI)  in  the  early  to 
mid-1990s.  By  design,  it  is  a  high-maturity 
development  process  with  all  the  features 
required  to  support  a  single  developer.  PSP 
is  a  measurement-driven  process  that 
includes  planning,  estimating,  design,  per¬ 
sonal  reviews,  and  testing.  Its  basic  concepts 
can  be  extended  for  aU  software  develop¬ 
ment  life-cycle  phases. 

The  Team  Software  Process™  (TSP'‘^“) 
was  developed  in  the  late  1990s  to  add  team- 
level  practices  to  the  PSP.  By  so  doing,  the 
TSP  makes  the  PSP  suitable  for  use  in  a 
commercial  software  development  environ¬ 
ment.  TSP  begins  with  a  facilitated  project 
launch  process  that  generates  a  detailed  proj¬ 
ect  plan.  The  project  plan  includes  a  devel¬ 
opment  strategy,  a  tailored  development 
process,  detailed  size  and  effort  estimates, 
earned  value  (EV)  plans,  a  schedule,  a  quali¬ 
ty  management  plan,  and  a  risk  management 
plan.  The  launch  process  is  a  team-building 
exercise  designed  to  foster  a  sense  of  own¬ 
ership  and  commitment  and  to  produce  a 
high-performance  work  team.  The  TSP  con¬ 
tinues  to  support  project  execution  activities 
via  a  structured  weekly  project  status  meet¬ 
ing  and  all  the  management  practices  neces¬ 
sary  to  run  a  full-scale  development  project. 

The  PSP  for  Engineers  course  is  taught 
in  the  context  of  new  development  where 
the  students  are  asked  to  complete  a  series  of 
10  programming  assignments.  As  a  result, 
there  is  a  perception  that  PSP  and  TSP  will 
only  work  for  new  development  projects.  In 
this  article,  we  will  demonstrate  how  PSP 
and  TSP  were  adapted  to  successfully  plan 
and  manage  a  maintenance  project. 

The  Team 

The  Maintenance  Team  Dilemma 

Maintenance  projects  deal  with  post-devel¬ 
opment  support.  In  general,  this  can  include 
operational  support  and  implementation  of 

SM  'P5P  and  PSP  are  service  marks  of  Carnegie  Mellon 
University. 


new  or  enhanced  features  as  well  as  defect 
investigations  and  fixes.  Some  maintenance 
projects  just  involve  working  off  a  backlog 
of  problem  reports  and  change  requests. 
These  projects  are  relatively  easy  to  handle  in 
a  conventional  TSP  launch.  Although  it  may 
be  necessary  to  modify  the  estimation  algo¬ 
rithm  because  there  is  not  a  good  correlation 
between  effort  and  the  size  of  the  change, 
there  is  a  known  list  of  tasks  (the  back- 
logged  problem  reports  and  change 
requests)  that  can  provide  the  basic  input  to 
the  TSP  planning  process. 

**When  preparing  to 
launch  a  TSP  team,  it  is  a 
good  practice  to  start  by 
identifying  a  successful 
end  state.  What  would 
management,  the  team, 
and  the  coach  consider 
to  be  a  successful  launch 
and  project?^* 


The  situation  gets  more  complicated 
when  near  real-time  operational  support  is 
added  to  the  mix  of  project  tasks.  The 
majority  of  project  tasks  may  be  unknown  at 
the  time  of  the  launch  because  the  opera¬ 
tional  anomalies  have  not  even  taken  place 
yet.  Nonetheless,  the  organization  needs  to 
allocate  adequate  resources,  commit  to 
scheduled  completion  dates  for  backlogged 
tasks,  and  manage  the  overall  effort. 

Organizations  often  have  difficulty  plan¬ 
ning,  staffing,  and  managing  these  sorts  of 
maintenance  projects  due  to  the  unpre¬ 
dictability  of  both  the  rate  at  which  opera¬ 
tional  anomalies  are  reported  and  the  effort 
required  to  respond  to  anomaly  reports. 
There  is  frequently  no  correlation  between 


the  length  of  an  anomaly  investigation  and 
the  size  of  the  resulting  change.  In  fact, 
many  anomaly  reports  do  not  even  result  in 
a  change. 

As  a  result,  team  members  frequently 
feel  that  any  attempt  to  estimate  and  plan  the 
effort  for  a  maintenance  project  is  futile. 
And  yet,  these  projects  need  to  be  staffed 
and  managed. 

This  article  shows  how  PSP  and  TSP 
were  successfully  used  to  plan  and  manage  a 
maintenance  project  and  how  the  team  was 
able  to  build  a  useful  estimating  scheme  in  a 
project  environment  where  it  was  common 
for  developers  to  spend  75  percent  of  their 
time  on  unplanned  event-driven  anomaly 
investigations. 

The  Project 

The  project  involved  maintaining  a  large  net¬ 
work-based  financial  services  package  origi¬ 
nally  implemented  in  C.  The  package  was 
key  to  the  company’s  revenue  stream  and 
required  real-time  operational  support  as 
well  as  fixes  and  enhancements. 

The  maintenance  team  was  composed  of 
12  PSP-trained  software  developers.  Half 
the  maintenance  team  was  located  at  a  site  in 
the  United  States,  while  the  remaining  half 
was  located  at  a  site  in  Europe.  There  was  a 
team  lead  at  each  location.  The  project  man¬ 
ager  worked  in  the  United  States.  The  TSP 
launch  was  held  in  the  team’s  US.  office. 

Is  It  a  Team? 

Typical  work  for  this  group  of  developers 
includes  investigating  reports  of  operational 
anomalies,  upgrading  legacy  software  to  sup¬ 
port  new  requirements,  adding  new  features, 
investigating  defects,  and  fixing  defects.  In 
the  past,  each  developer  considered  himself 
or  herself  responsible  for  a  different  prod¬ 
uct.  As  it  turned  out,  they  were  each  respon¬ 
sible  for  a  component  of  a  single  product. 
Each  product  component  was  built  and 
released  separately.  There  were  no  task  or 
schedule  dependencies  between  developers. 
Defects  were  generally  localized  to  a  com¬ 
ponent  of  the  product.  Up  to  this  point, 
individuals  only  needed  to  be  focused  on 
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their  own  component 

This  contrasts  strongly  with  the  situation 
in  new  development  where  there  are  usually 
obvious  dependencies  in  each  person’s  work 
on  tasks  being  done  by  other  team  members 
as  the  team  works  together  to  produce  a 
product. 

Watts  Humphrey  defines  the  functional 
team  as  a  team  with  the  following: 

...  has  a  functional,  rather  than  a 
product,  mission.  While  all  the  mem¬ 
bers  may  do  similar  work,  they  do 
not  develop  a  single  product  and 
their  individual  tasks  are  usually  quite 
independent.  . . .  [An]  example  would 
be  a  maintenance  group  where  each 
member  handles  the  repair  and 
enhancement  of  a  product.  While 
several  of  the  members  might  occa¬ 
sionally  work  on  elements  to  be  inte¬ 
grated  into  a  common  release,  they 
would  usually  work  alone.  [1] 

The  developers  on  this  project  fit  the 
definition  of  a  functional  team  perfectly. 
Consequendy,  the  decision  was  made  to 
launch  the  team  using  the  SEFs  variant  of 
the  TSP  launch  process  for  functional  teams, 
TSPf'. 

Preparation 

When  preparing  to  launch  a  TSP  team,  it  is 
a  good  practice  to  start  by  identifying  a  suc¬ 
cessful  end  state.  What  would  management, 
the  team,  and  the  coach  consider  to  be  a  suc¬ 
cessful  launch  and  project?  Answering  these 
questions  requires  having  well-defined 
launch  and  project  goals  and  a  strategy  for 
attaining  those  goals. 

Questions 

What  were  the  project  goals?  Why  was  this 
group  of  individual  developers  being 
launched  as  a  team?  What  did  the  organi¬ 
zation  or  the  project  manager  think  they 
would  be  able  to  accomplish  as  a  team  that 
they  would  not  be  able  to  accomplish  as 
individuals? 

We  drilled  down  on  these  questions  in  a 
series  of  launch  planning  meetings  with  the 
project  manager.  Ultimately,  the  project 
manager  was  able  to  clearly  articulate  three 
very  specific  objectives  that  provided  a  com¬ 
pelling  reason  for  these  individuals  to  work 
together  as  a  team: 

•  Spread  knowledge  among  the  team  and 
broaden  experience  to  reduce  areas  of 
risk  and  increase  efficiency. 

•  Improve  real  and  perceived  quality  to 
reduce  customer-generated  interrup¬ 
tions. 

•  Increase  ratio  of  planned  tasks  to  reac¬ 
tive  unplanned  activities. 


If  they  were  able  to  accomplish  these 
objectives,  it  would  benefit  not  only  the 
organization,  but  the  team  members  as  well. 

Once  we  had  a  well-defined  set  of  man¬ 
agement  objectives  for  the  team,  the  next  set 
of  questions  addressed  the  actual  TSP 
launch.  What  were  some  ways  that  the  team 
could  plan  for  the  unplanned,  high  priority 
interruptions  that  are  characteristic  of  near 
real-time  operational  support?  What  might 
they  use  as  a  size  metric?  Was  there  a  con¬ 
ceptual  design?  What  does  quality  mean 
when  they  are  modifying  a  small  number  of 
lines  relative  to  the  size  of  the  base  code? 

The  coach’s  goal  is  not  to  decide  the 
answers  to  these  questions  for  the  team,  but 
rather  to  consider  some  of  the  possible 
answers  and  to  make  sure  that  the  team  col¬ 
lected  the  right  project  data  prior  to  the 
launch  so  they  would  be  able  to  make  deci¬ 
sions  based  on  data  during  the  launch. 

Through  this  exercise  of  strategizing 
an  approach  for  conducting  the  launch,  it 
became  apparent  that  there  were  two 
overriding  themes:  commonality  and 
repeatability. 

Preparing  the  Project  Manager 

Commonality  was  the  theme  that  would  help 
the  individuals  to  gel  as  a  team.  If  they 
found  that  they  had  enough  in  common 
with  one  another  and  that  they  could  bene¬ 
fit  by  taking  advantage  of  the  synergy,  then 
surely  they  would  come  together  as  a  team. 

It  was  the  program  manager’s  job  to 
define  the  management  goals  that  would 
help  set  the  stage  for  the  team  to  gel.  These 
goals  had  to  serve  two  functions:  setting  a 
long-term  vision  for  the  project  and  the 
organization  while  providing  short-term  tar¬ 
gets  to  help  the  team  focus  and  come 
together. 

We  decided  that  the  best  approach 
would  be  to  identify  a  long-term  vision  to 
provide  a  frame  of  reference.  To  support  the 
vision,  a  series  of  short-term  goals  were 
developed.  The  team  would  be  given  three 
to  four  short-term  goals  with  the  expecta¬ 
tion  that  they  would  be  able  to  achieve  them 
within  about  a  month.  The  first  set  of  short¬ 
term  goals  was  selected  so  that  they  would 
be  achievable  with  relatively  low  risk. 

Since  maintenance  work  can  be  difficult 
to  predict  and  plan,  we  decided  that  the  team 
would  have  a  mini  re-launch  every  four  to 
six  weeks.  Each  of  these  re-launches  would 
provide  the  program  manager  an  opportuni¬ 
ty  to  roll  out  the  next  set  of  goals  on  the 
path  to  achieving  his  overall  vision. 

Preparing  the  Team  Members 

A  necessary  ingredient  to  this  team’s  success 
was  to  get  the  team  members  to  believe  that 
there  was  a  considerable  amount  of  repeata¬ 


bility  in  their  work  and  that  this  repeatability 
would  lead  to  an  improved  ability  to  esti¬ 
mate.  We  asked  the  team  members  to  start 
gathering  PSP  data  on  their  tasks  several 
weeks  prior  to  the  launch  in  an  attempt  to 
find  the  repeatability  in  their  daily  or  weekly 
activities. 

We  found  that  the  developers  had  two 
distinct  categories  of  tasks.  The  first  cate¬ 
gory  was  high  priority  interruptions. 
These  interrupts  could  occur  at  any  time. 
When  they  did,  aU  other  work  had  to  be 
put  aside.  The  second  category  of  task 
was  background  work,  which  consisted  of 
the  tasks  that  the  developers  worked  on 
when  they  were  not  reacting  to  high  prior¬ 
ity  interruptions. 

It  was  obvious  that  the  developers 
would  not  be  able  to  anticipate  what  inter¬ 
ruptions  would  occur.  However,  it  seemed 
that  there  might  be  a  pattern  to  the  quan¬ 
tity  of  high  priority  interrupt  effort  that 
was  spent  within  a  period  of  time.  If  the 
developers  could  quantify  the  percentage 
of  time  each  week  that  was  spent  handling 
the  high  priority  interrupts,  then  they 
would  be  able  to  create  a  budget  of  task 
hours  for  this  category  of  activity. 

The  developers  were  asked  to  flag  each 
task  they  worked  on  as  one  that  they  either 
knew  about  at  the  beginning  of  the  week  or 
one  that  came  in  during  the  week  as  an 
interrupt.  While  the  percentage  by  category 
differed  by  individual,  we  found  that  there 
was  clearly  repeatability  for  each  individual 
from  week  to  week.  The  developers  were 
able  to  use  the  data  they  collected  during 
launch  meeting  No.  4  to  plan  for  time  to  be 
spent  on  planned  tasks  as  well  as  to  create  a 
budget  for  time  to  be  spent  on  unplanned 
tasks.  The  planned  tasks  would  then  be  esti¬ 
mated  and  planned  just  like  any  other  PSP 
task.  Whenever  an  interrupt  came  in,  the 
developer  would  estimate  it  and  plan  it. 
Time  for  the  interrupt  would  be  allocated 
from  the  budget  for  unplanned  tasks. 

The  Launch 

A  TSP  launch  consists  of  a  series  of  script¬ 
ed  meetings.  We  tailored  the  standard  TSPf 
meeting  scripts  by  employing  some  special 
techniques  designed  to  contribute  to  the 
success  of  the  team  by  reinforcing  the 
themes  of  commonality  and  repeatability. 

Tailoring  -  Management  and 
Team  Goals 

For  each  goal,  we  asked  the  team  why  it  was 
important  to  them  and  what  the  impact  to 
the  team  would  be  if  they  missed  it.  They 
were  being  asked  to  justify  the  need  for  each 
goal.  In  doing  so,  they  began  to  internalize 
the  importance  of  the  goals,  converging  on 
a  common  understanding  of  what  the  goals 
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Activity 

Entry  Criteria 

Exit  Criteria 

Problem 

Investigation 

•  Anomaly  report,  defect  report,  or 
improvement  request  with 
assigned  due  date  and  priority. 

•  Root  cause  identified  and  documented. 

•  Anomaly  report  closed  or  reassigned. 

•  Additional  defect  reports  or  improvement 
requests  opened  as  necessary. 

•  Solution  investigation  assigned  if 
needed. 

Solution 

Investigation 

•  Defect  report  or  improvement 
request  with  assigned  due  date 
and  priority. 

•  Authorized  solution. 

Problem  Fix 

•  Defect  report  or  improvement 
request  with  assigned  due  date 
and  priority. 

•  Authorized  solution  (optional). 

•  Problem  fixed. 

•  Defect  report  or  improvement  request 
has  test  pending  status. 

Conformance  Test 

•  Test  request  opened. 

•  Test  scripts  are  received  from 
exchange. 

•  Test  completed. 

•  Filled  scripts  sent  to  exchange  or  test 
cancelled. 

•  Test  results  published. 

System  Build 

•  System  build  request  opened. 

•  All  defect  reports  and 
improvement  requests  are  cross- 
referenced  to  the  build  request 
and  have  test  past  status. 

•  All  code  changes  in  the  build  are 
traceable  to  a  cross-referenced 
defect  report  or  improvement 
request. 

•  Working  system  with  required 
documentation. 

Table  1 :  Maintenance  Process 


really  meant.  The  team  decided  to  break 
down  their  goals  into  prioritized  tasks  to  be 
interleaved  with  their  product  tasks. 


Tailoring  -  Conceptual  Design 

The  team  did  not  think  of  themselves  as 
part  of  a  bigger  project  so  they  were  initial¬ 
ly  hesitant  to  spend  time  reviewing  the  con¬ 
ceptual  design 

We  asked  the  design  manager  to  project 
a  system  diagram  on  a  screen  and  lead  a  dis¬ 
cussion  in  which  the  J)erson  most  knowledgeable 
with  each  component  of  the  system  briefed 
the  others  on  the  size  of  the  component, 
interactions  with  other  components  of  the 
system,  and  areas  of  risk. 

This  was  the  first  time  that  the  team 
members  ever  had  the  opportunity  to  see 


Figure  1:  Anomaly  Investigation  Piffort 
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their  own  work  on  individual  package  com¬ 
ponents  in  an  overall  system  context.  The 
discussion  became  animated  and  the  team 
members  became  plysically  involved'^.  In  spite 
of  their  initial  reticence,  this  was  the  meeting 
where  individuals  started  to  come  together 
as  a  team. 

Tailoring  -  Process  Plan 

The  organizational  maintenance  process  had 
been  defined  prior  to  the  launch.  So,  the 
team  focused  its  discussion  on  defining 
explicit  entry  criteria,  exit  criteria,  and 
required  approvals  for  moving  from  one 
process  step  to  the  next.  The  team  was  able 
to  identify  gaps  in  the  defined  process  and  to 
recommend  modifications  to  address  the 
gaps.  The  resulting  maintenance  process  was 
relatively  simple  and  is  shown  in  Table  1 . 

The  source  of  the  problem  could  be  a 
reported  anomaly,  a  defect  report,  or  a 
request  of  new  functionality.  Anomaly 
reports  do  not  necessarily  require  product 
changes  to  be  resolved.  They  could  be  oper¬ 
ator  errors,  procedural  issues,  etc.  The  scope 
of  the  solution  analysis  depends  on  the  size 
of  the  required  change,  and  it  could  be  omit¬ 
ted  for  a  trivial  change.  Conformance  testing 
verifies  that  the  software  will  work  in  the 
exchange  environment.  It  requires  test 
scripts  recorded  from  exchange  feeds. 
Multiple  changes  are  aggregated  into  a  sys¬ 
tem  build  and  released  to  production. 


Table  2:  Anomaly  Hstimation  Table 


Duration  (minutes) 

Very  Small 

6 

Small 

18 

Medium 

59 

Large 

187 

Very  Large 

596 

Tailoring  -  Top  Level  Plan 

The  team  used  the  data  they  had  collected 
over  the  past  several  weeks  and  analyzed  the 
average  number  of  task  hours  per  week  that 
they  had  each  spent  on  planned  versus 
unplanned  tasks.  Using  this  data,  they  were 
able  to  confidently  plan  for  how  many  hours 


they  could  get  on  task  each  week,  budgeting 
a  percentage  of  those  hours  for  unplanned 
or  interrupt-driven  tasks. 

For  many  of  the  team  members,  approx¬ 
imately  75  percent  of  their  task  hours  were 
being  spent  on  interrupt-driven  tasks.  So, 
those  team  members  who  were  achieving  a 
total  of  12  hours  on  task  per  week  planned 
to  spend  three  hours  per  week  on  back¬ 
ground  tasks  and  budgeted  nine  hours  a 
week  for  interrupt-driven  tasks. 

Tailoring  -  Quality  Plan 

The  team  spent  considerable  time  discussing 
what  quality  meant  to  their  customer.  A  pre¬ 
dominant  concern  was  defects  that  were 
returned  to  the  team  by  the  customer  after 
an  initial  fix.  The  team  put  metrics  in  place 
to  explicitly  track  and  manage  the  following; 

•  Quantity  and  frequency  of  returned 
fixes. 

•  Defect  investigation  time. 

•  Defect  turnaround  time. 

Tailoring  -  Detailed  Plan 

One  of  the  goals  of  this  launch  was  to 
build  a  plan  in  which  the  team  would  close 
out  aU  of  their  high  priority  tasks.  The 
background  tasks  were  sequenced  so  that 
the  high-priority  tasks  would  be  completed 
first.  Multiple  EV  plans  were  generated  so 
that  management  would  have  visibility  into 
the  following: 

•  How  long  it  would  take  to  complete 
the  high  priority  tasks  given  the  level 
of  interrupts  that  the  team  was  experi¬ 
encing. 

•  How  much  backlog  work  the  team 
would  complete  in  six  weeks. 

•  How  long  it  would  take  to  work  off 
the  full  backlog  of  tasks. 

This  was  easily  accomplished  with  the 
help  of  an  automated  scheduling  tool. 

Post-Mortem 

The  participants  felt  that  “team  synergy  was 
improved,”  that  they  did  a  “good  job  of  bal¬ 
ancing  work  load,”  and  that  there  was 
“exposure  of  everyone  else’s  jobs”  with 
“good  participation  and  contribution  from 
everyone.”  One  participant  summarized  the 
experience;  “This  seems  like  the  birthday  of 
this  team.” 

Results 

Estimating 

During  the  launch,  the  team  was  highly 
skeptical  about  their  ability  to  estimate 
anomaly  investigation  tasks.  However,  they 
agreed  to  make  their  best  estimates  and  then 
to  develop  an  estimating  algorithm  from  the 
post-launch  data.  Without  historical  data, 
the  team  estimating  error  was  41  percent. 
With  the  data  gathered  during  the  launch 
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post-mortem,  the  team  was  able  to  reduce 
the  estimating  error  to  23  percent. 

Anomaly  Investigation  Effort 

After  the  launch,  the  team  collected  data 
on  the  durations  of  anomaly  investigation 
tasks  for  several  months.  A  histogram’  of 
the  data,  as  shown  in  Figure  1,  indicates 
that  the  duration  of  anomaly  investigation 
tasks  can  be  modeled  as  a  random  variable 
and  that  a  skewed-distribution  function, 
like  a  log  normal,  would  provide  a  good 
basis  for  estimating  the  probable  duration 
of  an  anomaly  investigation  task. 

From  the  fitted  distribution  function,  it 
is  possible  to  determine  what  the  expected 
duration  of  a  very  small,  small,  medium, 
large,  and  very  large  task  would  be,  as 
shown  in  Table  2.  The  average  length  of  an 
investigation  task  was  59  minutes,  with 
approximately  70  percent  of  all  investiga¬ 
tions  requiring  between  18  and  187  minutes. 

Estimating  Algorithm 

This  result  leads  directly  to  a  simple  tech¬ 
nique  for  estimating  anomaly  investigations 
that  can  be  employed  during  a  TSP  launch. 
For  the  tasks  that  are  part  of  the  backlog 
for  a  special  anomaly  investigation,  catego¬ 
rize  each  one  as  very  small,  small,  medium, 
large,  or  very  large  and  use  the  estimated 
time  provided  by  the  table.  Prorate  the  total 
for  aU  backlogged  tasks  to  account  for  the 
unplanned  investigations  by  using  the  his¬ 
torical  percentage  of  time  devoted  to 
unplanned  anomaly  investigations.  For  this 
team,  the  percentage  of  time  devoted  to 
unplanned  anomaly  investigations  was  75 
percent  to  85  percent  of  its  available  time. 

Re-Estimating  the  Launch 

During  the  launch,  the  team  had  estimat¬ 
ed  the  effort  of  each  identified  investiga¬ 
tion  as  small,  medium,  or  large.  Assess¬ 
ment  of  effort  had  been  made  by  the  task 
owner,  based  on  familiarity  with  the  func¬ 
tionality  and  amount  of  code  that  would 
need  to  be  reviewed. 

During  post-mortem,  the  original 
small/medium/large  estimate  from  the 
launch  was  used  along  with  the  calculated 
values  of  small,  medium,  and  large  to  re- 
estimate  the  tasks.  As  shown  in  Table  3,  the 
re-estimate  based  on  the  calculated  size 
ranges  reduced  the  estimation  error  by  a 
factor  of  two  to  a  total  error  of  23  percent. 

Lessons  Learned 

The  coach  must  prepare  for  every  launch. 
The  coach  needs  to  understand  the  pecu¬ 
liarities  of  each  team,  anticipate  potential 
trouble  spots  relative  to  planning,  and 
have  a  strategy  for  how  to  facilitate  the 
launch  to  engage  the  participants  and  to 


Actual  Time 

Est.  Time  (Launch) 

Est.  Time  (S/M/L) 

Minutes 

2,549 

4,301 

3,313 

%  Error 

-41% 

-  23% 

Table  3:  Comparisons  of  estimates 


build  a  plan  that  supports  the  business 
objectives.  Choose  no  more  than  three  to 
four  very  specific  and  achievable  goals  to 
help  unite  the  participants  as  a  team.  Use 
PSP  1 .0  to  collect  data  prior  to  the  launch, 
and  focus  the  team  on  actual  data  to  help 
team  members  find  the  repeatability  in 
their  work  and  to  make  fact-based  deci¬ 
sions  during  the  launch. 

Even  though  an  individual  task  may  be 
completely  unpredictable,  once  the  statis¬ 
tics  that  characterize  a  set  of  typical  tasks 
are  known,  it  is  possible  to  use  those  sta¬ 
tistics  to  make  reasonably  accurate  esti¬ 
mates  about  the  effort  required  to  handle 
the  normal  workload  associated  with 
many  unpredictable  tasks.  This  allows  a  team 
to  allocate  the  tight  number  of  resources 
to  meets  its  commitments  and  bring  a 
sense  of  control  and  predictability  into  an 
apparentiy  chaotic  project.^ 
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Notes 

1 .  TSPf  is  still  in  the  prototype  stage,  but 
the  SEI  authorized  piloting  its  use  with 
this  team. 

2.  Our  experience  has  been  that  there 
comes  a  point  in  every  successful 
launch  where  the  team  members  get 
physically  involved  in  the  meetings. 
They  sit  up  straighter  in  their  chairs, 
leaning  forward  to  hear  better,  or  stand 
up  and  move  chairs  from  the  back  of 
the  room  to  the  front  so  that  they  can 
see  better  or  come  forward  to  write  on 
the  white  board. 

3.  The  histogram  shows  the  number  of 
data  samples  falling  into  each  of  the 
duration  bins  indicated  on  the  x-axis 
(bar  chart)  and  the  cumulative  number 
of  data  samples  with  duration  less 
than  the  x-axis  value  (curve).  For 
example,  there  are  83  anomaly  investi¬ 
gation  tasks  with  duration  between 
three  and  111  minutes  and  approxi¬ 
mately  65  percent  of  the  data  points 
have  a  duration  less  than  111  minutes. 
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David  P.  Quinn 
Borland  Software  Corporation 

In  the  struggle  to  set  expectations  and  reward  good  performance,  organisations  sometimes  tie  project  measures  to  performance 
incentives.  To  avoid  the  pitfalls  sometimes  caused  bj  this  link,  we  must  explore  the  nature  of  project  measures  and  perform¬ 
ance  and  how  to  possibly  link  project  measures  to  incentives  so  the  intended  behavior  occurs. 


Any  measure  when  viewed  in  isolation 
is  open  to  misinterpretation  and  mis¬ 
use.  Without  the  full  context  of  other 
measures,  we  may  unjustly  attribute  good 
or  poor  performance  based  on  viewing  a 
single  measure  as  an  absolute.  This  is  why 
linking  incentives  to  individual  measures 
can  be  detrimental  to  what  the  organiza¬ 
tion  is  trying  to  accomplish.  People  tend 
to  focus  on  meeting  a  measure  instead  of 
accomplishing  an  outcome.  People  tend  to 
focus  on  themselves  instead  of  the  team 
or  the  organization,  causing  problems  for 
others  and  negatively  impacting  the  over¬ 
all  organization’s  performance.  In  fact, 
linking  incentives  to  measures  often  has 
some  unintended  consequences. 

Effort  Variance  as  an  Example 

Effort  variance  is  a  very  useful  measure 
when  used  for  project  management  as  an 
indicator.  It  can  show  problems  with  esti¬ 
mates,  processes,  project  scope,  and  per¬ 
formance.  These  problem  areas  have  a  dif¬ 
ferent  meaning  depending  on  whether 
effort  variance  is  indicating  less  effort 
expended  than  estimated  or  more  effort 
expended  than  estimated. 

In  the  case  of  less  effort  expended 
than  estimated  (e.g.,  100  hours  estimated 
but  only  80  expended  for  a  -20  percent 
effort  variance),  here  are  some  causes  of 
why  the  variance  may  have  occurred: 

•  Estimate 

•  Engineers  padded  their  estimates. 

•  Estimation  parameters  were 
wrong. 

•  Scope 

•  Lack  of  understanding  of  scope. 

•  Scope  reduced  but  estimate  not 
updated. 

•  Process 

•  Steps  skipped. 

•  A  process  improvement  occurred. 

•  Performance 

•  Hours  worked  were  not  record¬ 
ed/  reported. 

•  Brilliant  work  was  performed. 

•  Missed  one  or  more  requirements. 

•  Did  not  complete  the  task. 

•  Allowed  poor  quality  in  order  to 
meet  a  deadline. 


Of  these  causes,  only  two  (brilliant 
performance  and  process  improvement) 
should  result  in  a  reward  through  an 
incentive.  The  other  causes  should  result 
in  some  sort  of  remedial  action. 

On  the  other  hand,  when  looking  at 
more  effort  expended  than  estimated  (e.g, 
100  hours  estimated  but  120  expended  for 
a  +20  percent  effort  variance),  here  are 
some  causes  of  why  the  variance  may  have 
occurred: 

•  Estimate 

•  Someone  lowered  the  original  esti¬ 
mates  to  meet  mandated  cost/ 
schedule. 

•  Estimation  parameters  were 
wrong. 

*^The  biggest  potential 
problem  with  linking 
incentives  to  effort 
variance  is  that  people 
may  game  the  system.^* 

•  Scope 

•  Lack  of  understanding  of  scope. 

•  Scope  increased  but  the  estimate 
was  not  updated. 

•  Customer  was  indecisive  on  the 
requirements. 

•  Process 

•  Process  is  inefficient. 

•  The  process  does  not  match  the 
customer’s  needs. 

•  Unnecessary/inappropriate  steps 
were  taken. 

•  The  work  from  someone  earlier  in 
the  process  made  this  person’s 
work  more  difficult. 

•  Performance 

•  Meets  initial  estimate  but  does  not 
meet  the  modified  estimate  to  meet 
budget/schedule  (i.e.,  someone 
changed  the  estimate  without 
changing  scope  just  to  make  the 
numbers  match  the  preferred 


schedule). 

•  Poor  work  performance. 

•  Added  capability  the  customer  did 
not  request. 

As  with  the  previous  example,  not  all 
of  these  causes  should  detract  from  incen¬ 
tives.  Only  poor  work  performance  and 
using  the  wrong  parameters  are  signs  that 
the  engineer  needs  to  adjust  his/her 
behavior.  Many  of  the  others  are  organi¬ 
zational  or  managerial  faults  that  impact 
the  engineer’s  performance. 

Before  rewarding  or  punishing  for 
effort  variance,  there  are  other  factors  to 
consider.  Perhaps  it  is  okay  that  effort 
variance  is  above  estimate  if  the  quality  of 
code  leads  to  reduced  effort  variance  for 
testing  (i.e.,  there  is  no  real  impact  to 
schedule)  or  reduced  rework  to  correct 
defects.  Perhaps  the  minor  changes  in 
scope  (that  no  one  felt  required  changing 
in  the  estimates)  really  hit  harder  than 
thought  and  showed  up  in  the  effort  vari¬ 
ance.  You  would  not  know  if  the  latter 
case  was  true  unless  requirements  volatili¬ 
ty  measures  were  gathered. 

Other  Indicators 

The  impact  of  effort  variance  on  projects 
and  on  the  organization  must  be  examined 
to  get  an  adequate  picture  of  the  impor¬ 
tance  of  effort  variance.  For  instance,  a 
team  of  developers  may  take  shortcuts  to 
reduce  effort  variance,  but  the  shortcuts 
negatively  impact  quality.  That  may 
increase  the  effort  variance  of  the  test 
team  who  must  perform  more  test  cycles 
than  estimated.  The  developers  may  get 
rewarded  for  their  improved  effort  vari¬ 
ance  at  the  expense  of  the  test  team. 

Significant  effort  variance  should 
result  in  a  change  in  schedule  perform¬ 
ance.  If  team  members’  effort  variance  is 
-20  percent,  the  schedule  should  see  a 
comparable  variance  (i.e.,  the  schedule 
variance  should  be  roughly  -20  percent). 
When  there  are  major  gaps  between  the 
two  variances  (for  example,  -15  percent 
effort  variance  and  -5  percent  schedule 
variance),  this  should  signal  that  some¬ 
thing  may  be  wrong.  The  organization 
needs  to  investigate  why  the  effort  per- 
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Measure 

High 

Medium 

Low 

Unsatisfactory 

Effort 

Variance 

Any  variance 
better  than  -X 
percent. 

Variance  between 
-X  percent  and  -Y 
percent. 

Variance  between  -Y 
percent  and  +Z  percent. 

Variance  greater 
than  Z  percent. 

Schedule 

Variance 

Any  variance 
better  than  -X 
percent. 

Variance  between 
-X  percent  and  -Y 
percent. 

Variance  between  -Y 
percent  and  +Z  percent. 

Variance  greater 
than  Z  percent. 

Quality 

Any  variance 
better  than  X 
percent  of  the 
team  average. 

Variance  between 

X  percent  and  Y 
percent  of  the  team 
average. 

Variance  between  Y 
percent  and  Z  percent 
of  the  team  average. 

Defect  rate  worse 
than  Z  percent 
of  the  team  average. 

Process 

Compliance 

Recommended 
process 
improvement 
accepted  for 
implementaion 
and  complied  with 
defined  processes. 

Participated  in 
process 
improvement 
activities  and 
complied  with 
defined  processes. 

Complied  with  defined 
processes  on  regular 
basis. 

Inconsistent  use 
of  defined  processes. 

Assumptions  X,Y,  and  Z  may  be  different  for  each  measure. 

Quaiity  may  move  from  being  relative  to  team  members  to  being  organizationaiiy  reiative. 
More  measures  couid  be  added  if  desired. 


Table  1:  Example  Peiformance  Incentive  Structures 


formance  is  not  impacting  the  schedule 
more.  Perhaps  other  factors  inhibit  an 
improvement  in  schedule  variance. 

Other  Problems  With  Linkage 

There  are  other  problems  with  linking 
incentives  to  individual  measures.  Based 
on  the  organizational  learning  that  occurs 
from  doing  projects,  performance  that 
once  provided  a  bonus  for  people  will  no 
longer  result  in  a  bonus  as  the  organiza¬ 
tion  adjusts  process  performance  meas¬ 
ures.  The  following  provides  an  example 
for  effort  variance. 

One  of  the  reasons  to  gather  effort 
information  is  to  ensure  your  estimating 
model  is  correct.  If  team  members  show 
they  are  expending  far  fewer  hours  than 
estimated  on  a  regular  basis  (better  than 
—10  percent  effort  variance),  the  organiza¬ 
tion  should  update  the  estimation  model 
to  reflect  this  performance.  This  wiU  pro¬ 
vide  more  accurate  effort  estimates  in  the 
future.  However,  once  the  organization 
modifies  the  estimation  model,  team 
members  will  no  longer  qualify  for  incen¬ 
tives,  even  though  they  perform  at  the 
same  level  the  organization  rewarded 
before. 

But  not  updating  the  estimation  model 
is  wrong.  If  the  customer  continues  to  see 
that  effort  variance  is  significantly  below 
estimates,  the  customer  is  likely  to  assume 
that  engineers  are  padding  their  estimates. 
Trust  is  broken  and  the  customer  will 
require  that  the  organization  reduce  its 
effort  estimates  on  future  projects  no  mat¬ 
ter  how  reasonable  the  estimate  may  be. 
The  organization  must  update  the  estima¬ 
tion  model  despite  the  impact  on  the 
incentives. 

Performance  Is  Relative 

Just  as  one  measure  by  itself  may  not  teU 
the  whole  story,  sometimes  that  one  meas¬ 
ure  hides  the  truth.  While  other  measures 
may  give  a  more  accurate  picture  of  where 
performance  stands  as  a  whole,  measures 
may  indicate  where  an  individual  engineer 
stands  against  his/her  peers.  A  single 
measure  by  itself  may  indicate  that  an 
engineer  is  performing  well,  but  in  com¬ 
parison  to  other  engineers,  the  engineer’s 
performance  may  be  lacking. 

While  it  may  appear  logical  to  reward 
someone  who  has  a  —10  percent  effort 
variance,  it  does  not  make  sense  if  the 
organization’s  average  effort  variance  is 
—15  percent.  Technically,  this  person’s  per¬ 
formance  is  slowing  down  the  organiza¬ 
tion,  and  this  person  is  not  performing  up 
to  the  level  of  his/her  peers.  Likewise,  it 
may  be  good  to  reward  someone  with  a 
+5  percent  effort  variance  if  the  rest  of 


the  organization  is  performing  at  a  +12 
percent  effort  variance. 

There  is  a  problem  with  this  compari¬ 
son  though.  The  engineer  with  a  —1 0  per¬ 
cent  effort  variance  while  everyone  else 
has  a  —15  percent  effort  variance  may  be 
handling  all  the  tough,  complex  tasks. 
Comparing  performance  between  peers 
using  measures  like  effort  variance  is  not 
as  wise  as  it  may  appear. 

Other  Potential  Problems 

Someone  who  consistently  outperforms 
the  estimated  effort  does  something  dif¬ 
ferent  from  everyone  else.  If  that  person 
does  not  share  that  something  different 
with  other  people,  the  incentives  are  rein¬ 
forcing  the  wrong  behavior.  As  someone 
discovers  a  process  improvement  or  other 
type  of  improvement,  the  person  should 
share  that  information,  and  the  organiza¬ 
tion  should  reward  that  sharing.  Someone 
who  does  not  share  improvement  infor¬ 
mation  is  not  acting  in  the  organization’s 
best  interest. 

The  biggest  potential  problem  with 
linking  incentives  to  effort  variance  is  that 
people  may  game  the  system.  This  usual¬ 
ly  takes  two  forms:  people  work  extra 
hours  but  do  not  record  them,  or  people 
place  the  hours  worked  against  a  different 
activity.  In  the  latter  case,  these  hours  are 
not  necessarily  charged  to  non-biUable 
activities.  People  may  charge  the  hours  to 
activities  on  other  projects  with  available 
budget. 

In  either  case,  the  organization  is  not 
getting  accurate  information  on  what  it 
takes  to  get  a  project  done.  The  organiza¬ 
tion  is  no  longer  able  to  learn  how  accu¬ 
rate  the  estimates  are  or  if  a  process 
improvement  occurred.  Organizations 
that  find  people  gaming  the  system 
because  of  the  linkage  between  incentives 
and  measures  usually  have  to  take  one  of 


two  solutions.  The  first  is  to  remove  the 
linkage.  The  second  is  to  have  a  policy  that 
makes  entering  inaccurate  effort  data  a 
cause  for  dismissal.  Organizations  tend  to 
choose  the  former  rather  than  the  latter 
because  the  organization  does  not  want  to 
lose  good  people. 

Recognizing  and  Rewarding 
Outstanding  Performance 

Project  performance  measures  can  be  a 
contributor  to  recognizing  outstanding 
performance,  but  there  should  not  be  a 
direct  correlation  between  a  single  meas¬ 
ure  and  an  incentive.  A  measure  that 
makes  up  X  percent  of  an  incentive  (no 
matter  how  small  X  is)  is  more  likely  to  be 
gamed,  destroying  any  chance  to  accurate¬ 
ly  measure  organizational  performance.  A 
better  approach  would  be  to  group  a  num¬ 
ber  of  measures  together  and  reward 
them  as  an  overall  performance. 

For  instance,  an  organization  may  rate 
a  software  engineer  on  effort,  schedule, 
quality,  and  process  compliance  (see  Table 
1).  The  organization  may  rate  each  factor 
as  high,  medium,  low,  or  unsatisfactory 
(see  Table  2).  The  combination  of  highs. 

Table  2:  Compensation  Formulas 


Highs 

Mediums 

Lows 

Unsatisfactory 

Percent 

Compensation 

4 

0 

0 

0 

100% 

3 

0 

0 

95 

3 

0 

1 

0 

90 

2 

2 

0 

0 

90 

2 

1 

0 

80 

2 

0 

2 

0 

75 

1 

3 

0 

0 

75 

1 

2 

1 

0 

60 

0 

4 

0 

0 

50 

1 

2 

0 

50 

0 

3 

1 

0 

25 

1 

0 

3 

0 

25 

0 

2 

2 

0 

15 

0 

3 

0 

10 

0 

0 

4 

0 

0 

- 

- 

1  or  more 

0 
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mediums,  and  lows  indicate  the  incentive 
reward.  Getting  a  high  rating  in  all  four 
factors  would  get  the  maximum  incentive. 
Getting  three  high  ratings  and  one  medi¬ 
um  rating  would  get  a  certain  percentage. 
The  organization  would  further  adjust 
incentives  for  different  combinations  of 
highs,  mediums,  and  lows.  Any  unsatisfac¬ 
tory  rating  would  automatically  eliminate 
the  incentive. 

This  helps  the  engineer  understand 
that  each  of  these  measures  is  important 
but  no  one  measure  is  more  important 
than  the  other  measures.  It  also  allows  for 
the  fact  that  the  measures  are  interdepend¬ 
ent  and  focuses  on  how  the  organization 
views  technical  performance  as  a  whole. 

Personal  Experience  With 
Unintended  Consequences 

I  have  two  sons,  David  and  Mark.  Like 
most  young  boys,  they  constantly  want 
raises  in  their  allowances.  I  am  a  firm 
believer  that  an  increase  in  wages  results 
from  an  increase  in  responsibility.  I  also 
am  a  firm  believer  in  trying  to  get  the 
boys  to  do  chores  I  do  not  find  entirely 
enjoyable.  Therefore,  I  needed  to  devise  a 
way  to  allow  them  to  earn  more  money 
while  making  my  life  easier. 

I  targeted  mowing  the  lawn,  a  chore  I 
dislike.  Unfortunately,  they  were  too 
young  to  mow  the  lawn.  However,  before 
I  mow  the  lawn  each  week,  there  is  anoth¬ 
er  necessary  chore.  We  have  two  dogs: 
Ace,  a  retired  racing  greyhound,  and 
Amigo,  a  longhaired  Chihuahua.  Their 
daily  routine  generates  a  set  of  piles  that 
someone  must  gather  before  I  can  mow 
the  backyard.  That  became  their  chore. 

David,  the  older  son,  gets  $2  for  clean¬ 
ing  the  piles  in  the  backyard.  When  David 
is  done,  Mark  gets  $1  to  find  any  missed 
piles.  However,  I  add  an  incentive  to  this. 
For  every  Ace  pile  Mark  finds,  Mark  gains 
25  cents  and  David  loses  25  cents.  For 
every  Amigo  pile  Mark  finds,  Mark  gains 
10  cents  and  David  loses  10  cents.  David 
cannot  go  below  $1  total  but  Mark  has  a 
Umitless  incentive.  When  Mark  finishes  his 
chore,  I  do  a  final  inspection  of  the  back¬ 
yard.  As  with  David,  Mark  loses  25  cents 
for  every  Ace  pile  I  find  and  loses  10  cents 
for  every  Amigo  pile  I  find.  It  seemed  like 
a  great  system. 

One  spring  I  sent  David  out  to  do  his 
task.  I  lost  track  of  Mark  shortly  after 
David  started  but  found  him  shortly 
before  David  was  done.  Since  it  was  start¬ 
ing  to  get  dark,  I  started  Mark  on  his 
chore  while  David  finished  the  last  third 
of  the  yard. 

As  they  both  continued  their  chores. 


David  suddenly  announced  that  one  of 
the  Ace  artifacts  was  covered  with  grass.  I 
did  not  think  much  of  it  until  he 
announced  a  few  seconds  later  that  other 
artifacts  were  covered  by  grass.  Mark 
commented  that  more  artifacts  are  likely 
covered  by  grass.  I  was  able  to  put  two 
and  two  together  and  pointedly  asked 
Mark  if  he  had  been  covering  up  the  arti¬ 
facts.  His  face  turned  white,  his  jaw 
dropped,  and  he  meekly  let  out  a  “Yes.” 

I  immediately  sent  him  in  the  house  to 
get  his  bath  and  go  to  bed.  I  also  let  him 
know  he  forfeited  his  money  for  doing 
the  chore.  As  soon  as  the  door  closed,  I 
laughed  so  hard  I  thought  I  would  cry.  I 
did  not  realize  my  performance  incentives 
would  create  that  p'pe  of  behavior.  You 
really  have  to  be  careful  when  linking  per¬ 
formance  measures  to  incentives. 

Conclusion 

Gathering  and  using  measures  is  an  essen¬ 
tial  part  of  business.  Measures  provide 
outstanding  insight  into  current  status, 
possible  problems,  and  process  improve¬ 
ments.  However,  one  measure  does  not 
provide  enough  insight  by  itself  Measures 
must  be  viewed  in  total  as  the  result  of 
one  measure  may  impact  another  meas¬ 
ure.  The  data  gathered  for  the  measures 
must  be  unquestionably  accurate.  When 
organizations  tie  incentives  to  project 
measures,  people  often  report  the  data 
inaccurately  to  meet  the  incentives.  Once 
organizations  realize  data  is  inaccurate, 
they  tend  to  break  the  link  between  incen¬ 
tives  and  measures.  They  recognize  that 
data  needs  surpass  the  incentive  needs. ♦ 
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Welcome  to  “Backfire,”  the  interview 
within  a  journal  where  we  cross- 
examine  popular  icons  for  software  truth. 
This  month  we  have  the  cast  of  Seinfeld  - 
Jerry,  George,  Elaine,  and  Kramer  who  have 
developed  a  Comedy  Maturity  Model 
(CMM)  to  teach  young  comedians. 

Gary:  First  things  first,  what  gave  you  the 
idea  to  teach  comedy  to  the  masses? 

Jerry:  Everybody  is  so  serious  now  days.  I 
thought  to  myself,  what  is  going  on  in  this 
community?  Are  you  people  aware  of  what 
is  happening?  What  is  driving  you  to  this 
behavior?  Is  it  the  humidity?  Is  it  the 
Muzak?  Is  it  the  white  shoes? 

George:  That  is  so  true!  I  have  no  funny 
friends.  I  am  the  funny  one  —  El  Clowno! 

Gary:  So  you  formed  a  company. 

Kramer:  Yes,  the  company  is  Somewhat 
Comical  Institute  (SCI),  pending  legal 
review. 

Jerry:  I  preferred  the  name  Super  Silly  Inc. 
George:  Elaine  wanted  the  name  to  be  Silly 
Putty  Limited,  for  obvious  reasons,  but 
there  were  trademark  issues. 

Gary:  Who  came  up  with  the  idea  of  creat¬ 
ing  a  Comedy  Maturity  Model? 

Jerry:  Kramer.  He  was  watching  a  PBS  spe¬ 
cial  on  software. 

George:  I  love  those  people.  You  can’t  ask 
them  questions.  They  are  so  mentally  gifted 
that  we  must  not  disturb  the  delicate  genius 
unless  it  is  in  the  confines  of  an  office. 
When  huge  sums  of  money  are  involved 
then  the  delicate  genius  can  be  disturbed. 
Kramer:  I  come  up  with  these  things,  I 
know  they  are  gold  but  nothing  happens, 
hence,  I  called  Elaine. 

Gary:  Why  Elaine? 

Kramer:  She’s  a  calculating,  cold-hearted 
businesswoman.  When  there  is  dirty  work  to 
be  done,  she  doesn’t  mind  stomping  on  a 
few  throats. 

Gary:  Why  not  George  or  Jerry? 

Kramer:  There  is  a  littie  too  much  chlorine 
in  that  gene  pool. 

George:  Yeah,  I’m  a  great  quitter.  It’s  one  of 
the  few  things  I  do  well.  I  come  from  a  long 
line  of  quitters.  My  father  was  a  quitter;  my 
grandfather  was  a  quitter  ...  I  was  raised  to 
give  up. 

Gary:  Tell  me  about  the  model. 


Elaine:  It  has  five  levels.  AH  good  models 
have  five  levels. 

George:  I  wanted  seven. 

Jerry:  The  first  level  is  “Breathing,”  every¬ 
one  qualifies  in  this  level  because  we  believe 
everyone  has  a  little  humor. 

George:  I  say  stupid  things  all  the  time.  I 
cannot  go  two  minutes  without  saying 
something  stupid. 

Kramer:  We  didn’t  want  to  exclude  poten¬ 
tial  clients  so  we  set  the  bar  low.  It’s  like  soc¬ 
cer  -  everyone  gets  to  play. 

Gary:  What  comes  after  the  breathing  level? 
Elaine:  Level  2  is  “Grin,”  the  basic  art  of 
remembering  jokes.  Level  3  is  “Giggle,” 
delivering  jokes.  Level  4  is  “Guffaw,”  the  art 
of  creating  jokes.  Level  5  is  “Gut  Buster,” 
stringing  jokes  into  a  witty  routine. 

George:  Each  level  has  several  KCAs. 

Gary:  KCAs? 

Jerry:  Knowledge  Comedy  Areas. 

Kramer:  We  load  the  lower  levels  with  a  lot 
of  banal  KCAs  and  then  reduce  them  to  a 
few  pedantic  KCAs  in  the  upper  levels.  That 
allows  us  to  hook  them  and  keep  them. 

Gary:  There  is  not  much  comedic  meat  to 
the  model? 

George:  I  think  that  you  think  that  a  certain 
something  is  not  all  that  it  could  be,  when, 
in  fact,  it  is  all  that  it  should  be  ...  and  more! 
Jerry:  You  have  to  be  patient  with  models;  it 
is  like  knocking  over  a  Coke  machine.  You 
can’t  do  it  in  one  push.  You  have  to  rock  it 
back  and  forth  a  few  times,  and  then  it  goes 
over. 

Elaine:  You  know  what  they  say,  “You 
don’t  sell  the  steak;  you  sell  the  sizzle.” 

Gary:  It  sounds  like  the  SCI  CMM  is  more 
about  making  money  than  helping  comedi¬ 
ans. 

George:  Why  would  we  want  to  help  some¬ 
body?  That  is  what  nuns  and  Red  Cross 
workers  are  for. 

Kramer:  It’s  like  the  Dewey  Decimal 
System  ...  what  a  scam  that  was.  I  could  raise 
enough  money  to  cure  polio. 

Jerry:  It’s  about  nothing,  a  model  about 
nothing. 

Gary:  How  will  you  make  money  on  the 
CMM? 

Kramer:  Assessments,  workshops,  and  con¬ 
sulting. 

Gary:  Cosmo  Kramer  is  consulting? 


Kramer:  Oh,  I’m  out  there  Gary,  I’m  out 
there! 

Gary:  Who  would  consider  you  comedy 
consultants? 

George:  Hey,  we  have  artistic  integrity. 
Jerry:  Artistic  integrity?  Where  did  you 
come  up  with  that?  You  are  not  artistic  and 
you  have  no  integrity. 

George:  You  know,  if  you  take  everything  I 
have  done  in  my  entire  life  and  condense  it 
down  into  one  day,  it  looks  decent. 

Kramer:  The  real  money  is  in  the  spin-offs 
baby! 

Gary:  Spin-offs? 

Elaine:  Oh  yes  we  have  the  Stand-up 
Comedy  Maturity  Model  (Sup-CMM), 
Improvisation  Comedy  Maturity  Model  (I- 
CMM),  Situation  Comedy  Maturity  Model 
(Sit-CMM),  Political  Comedy  Maturity 
Model  (P-CMM),  and  for  the  night  owls  the 
Late  Night  Comedy  Mamrity  Model  (Ln- 
CMM). 

Gary:  Will  all  the  spin-offs  confuse  young 
comedians? 

Jerry:  Of  course,  but  we  will  integrate  the 
models  and  sell  services  to  understand  the 
new  integrated  model. 

Gary:  It  seems  like  a  long  road  just  to  get 
back  to  one  model. 

Jerry:  The  road  less  traveled  is  less  traveled 
for  a  reason. 

George:  It’s  like  selling  them  a  car,  you  stick 
them  with  the  undercoating,  rust  proofing, 
dealer  prep  ...  suddenly  they  are  on  their 
backs  like  turtles. 

Gary:  So,  it  is  a  He. 

George:  Just  remember,  it’s  not  a  He  if  you 
beUeve  it. 

Gary:  You’re  crazy. 

Kramer:  Are  we,  or  are  we  so  sane  that  you 
just  blew  your  mind? 

Jerry:  Maturity  models  are  funny  business. 

Okay,  thank  you  Jerry,  George,  Elaine, 
and  Kramer.  Join  us  next  time  when 
Backfire  cross-examines  Butch  Cassidy  and 
the  Sundance  Kid  on  outsourcing. 

—  Gary  A.  Petersen 

Shim  Enterprise,  Inc. 
gary.petersen@shiminc.com 
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to  give  every  project  its  best  chance  in  an  industry 
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